Tuesday, November 16, 2010

Fortran program to calculate an integral using "crude" Monte Carlo

The following program calculates the integral I

using "crude" Monte Carlo. In this case, n=3, xia=0, xib=1 and f(x)=exp(a·x).


Program intmc

double precision fsum,fsum2,a,xa,xb,V,x,f,I,IA,varf
integer M,n,j,k
dimension a(3),xa(3),xb(3),x(3)
c n: dimension of the array x
c M: number of random numbers we use
c V: volum of the hipervolume
n=3
M=1000000
V=1
do j=1,n
a(j)=rand()
xa(j)=0
xb(j)=1
V=V*(xb(j)-xa(j))
enddo
c let's find the value of the integral that we get using monte-carlo
fsum=0
c fsum2 will help us to find the error
fsum2=0
do k=1,M
do j=1,n
x(j)=rand()
enddo
call func(a,x,f,n)
fsum=fsum+f
fsum2=fsum2+f**2
enddo
c I is the value of the integral we wanted to find
c IA is the value of the integral calculated analiticaly
I=V*fsum/M
varf=(fsum2/M-((fsum/M)**2))
sigma=sqrt(varf/M)
IA=1
do j=1,n
IA=IA*((exp(a(j))-1)/a(j))
enddo
write(*,*) I,sigma*3,IA
stop
end


SUBROUTINE func(a,x,f,n)
double precision a,x,f,ax
integer n
dimension a(n),x(n)
ax=0
do j=1,n
ax=ax+a(j)*x(j)
enddo
f=exp(ax)
end

Fortran program to calculate an integral using the method of trapezes

The following program calculates the integral I

using the trapezes method. In this case, n=3, xia=0, xib=1 and f(x)=exp(a·x).


Program inttr
double precision isum,x,xa,xb,h,f,I,a,IA,error,e
integer j,p,k1,k2,k3,n,l
dimension x(3),xa(3),xb(3),h(3),a(3)
n=3
p=300
do j=1,n
a(j)=rand()
xa(j)=0
xb(j)=1
h(j)=(xb(j)-xa(j))/p
enddo
isum=0
error=0
c we have to use n do's one inside the other. In this case, n=3.
c to calculate the error we need to compute the second partial
c derivate of F
do k1=1,p
x(1)=xa(1)+(k1-1/2)*h(1)
do k2=1,p
x(2)=xa(2)+(k2-1/2)*h(2)
do k3=1,p
x(3)=xa(3)+(k3-1/2)*h(3)
call func(a,x,f,n)
isum=isum+f
do l=1,n
error=error+f*(a(l)**2)
enddo
enddo
enddo
enddo
I=isum*(h(1)*h(2)*h(3))
e=(p**(-2/n))*error/(24*isum)
IA=1
do j=1,n
IA=IA*((exp(a(j))-1)/a(j))
enddo
write(*,*) I,e,IA
stop
end



SUBROUTINE func(a,x,f,n)
double precision a,x,f,ax
integer n,j
dimension a(n),x(n)
ax=0
do j=1,n
ax=ax+a(j)*x(j)
enddo
f=exp(ax)
end

Metropolis Algorithm

A very interesting link where we can understand easily the Metropolis Algorithm:

http://www.childrens-mercy.org/stats/weblog2007/MetropolisAlgorithm.asp

Saturday, November 6, 2010

Calculate PI with a simple fortran program

In this link (Fer problemes), you can find a montecarlo problem that allows us to find the number PI.

Here, you have one code that could be implemented to simulate this program. It is done in fortran.

PROGRAM calculpi

integer b,d,R
real acirc, aquad
double precision div
b=12345
d=76545
R=1
acirc=0
aquad=0
do i=1,1000000
a=rand(b)
c=rand(d)
b=a
d=c
if ((a**2+c**2)
acirc=acirc+1
endif
if ((a
aquad=aquad+1
endif
div=(acirc/aquad)*4

if (mod(i,10)==0)then
write (*,*) acirc, aquad, div
endif
enddo
STOP
END

Friday, November 5, 2010

3D plots in gnuplot

The comand is:

gnuplot> splot "positions.dat"

Wednesday, November 3, 2010

Problem in fortran using lapack

Consider the linear system of equations Ax=b with the block structure:

We know that the LU decomposition of A is:

Make a program in fortran to compute the LU decomposition and to find the solution.

_________________________________________________________________



Program ludecomposition

integer INFO1,INFO2, i, j, N, N1,N2,N3
integer, ALLOCATABLE, dimension (:) :: IPIVZ1,IPIVZ2
double precision, ALLOCATABLE, DIMENSION(:,:) :: A11,A13,A22,
c A23,A31,A32,A33,S,Z3,Z4,A11C,A22C
double precision, ALLOCATABLE, DIMENSION(:) ::B1,B2,B3
c we let the user to enter the dimensions N1,N2 and N3.
open (unit=1, file="solution.dat")
write(*,*) 'Input the 3 values n1,n2,n3.After each press enter.'
read (*,*) N1,N2,N3
c The dimension of A is N. Let's compute N.
N=N1+N2+N3
C A is a square matrix. That's why we have to compute M=N.
ALLOCATE(A11(N1,N1),A13(N1,N3),A22(N2,N2),A23(N2,N3),
c A31(N3,N1),A32(N3,N2),A33(N3,N3),S(N3,N3),Z3(N3,N1),
c Z4(N3,N2),A11C(N1,N1),A22C(N2,N2))
ALLOCATE(B1(N1),B2(N2),B3(N3),IPIVZ1(N1),IPIVZ2(N2))
C Let's full the matrixes AII with random numbers.
do i=1,N1
do j=1,N1
A11(i,j)=rand()
enddo
do j=1,N3
A13(i,j)=rand()
enddo
enddo
do i=1,N2
do j=1,N2
A22(i,j)=rand()
enddo
do j=1,N3
A23(i,j)=rand()
enddo
enddo
do i=1,N3
do j=1,N1
A31(i,j)=rand()
enddo
do j=1,N2
A32(i,j)=rand()
enddo
do j=1,N3
A33(i,j)=rand()
enddo
enddo
C Now the matrixs AII are full of random numbers. In order to see if
C it works correcly, we can full the matrix b with the elements of
C each row added.
do i=1,N1
B1(i)=0
do j=1,N1
B1(i)=B1(i)+A11(i,j)
enddo
do j=1,N3
B1(i)=B1(i)+A13(i,j)
enddo
enddo
do i=1,N2
B2(i)=0
do j=1,N2
B2(i)=B2(i)+A22(i,j)
enddo
do j=1,N3
B2(i)=B2(i)+A23(i,j)
enddo
enddo
do i=1,N3
B3(i)=0
do j=1,N1
B3(i)=B3(i)+A31(i,j)
enddo
do j=1,N2
B3(i)=B3(i)+A32(i,j)
enddo
do j=1,N3
B3(i)=B3(i)+A33(i,j)
enddo
enddo
C now, we call our subroutine in order to compute the LU decomposition
C of the matrix A composed by our AII matrixes
call ludec(A11,A13,A22,A23,A31,A32,A33,
C N1,N2,N3,S,Z3,Z4,A11C,A22C,IPIVZ1,IPIVZ2)
C finally, we call our subrouine in order to get the values of x
call solve(A11,A13,A22,A23,A31,A32,A33,
C N1,N2,N3,S,Z3,Z4,B1,B2,B3,A11C,A22C,IPIVZ1,IPIVZ2)
C let's write the result of the computation. If we write B, we are
C writting the solution of the system.
write(1,*)'The solution of the x-array you wanted to know is'
do i=1,N1
write(1,*)B1(i)
enddo
do i=1,N2
write(1,*)B2(i)
enddo
do i=1,N3
write(1,*)B3(i)
enddo
write(*,*)'The solution can be found in the file solution.dat.'
C in our example, x-array should be a vector of ones.
stop
end

C==================================================
subroutine ludec(A11,A13,A22,A23,A31,A32,A33,
C N1,N2,N3,S,Z3,Z4,A11C,A22C,IPIVZ1,IPIVZ2)
integer INFO,i,j,k
CHARACTER(1) TRANSN,TRANST
integer IPIVZ1(N1),IPIVZ2(N2)
double precision A11,A13,A22,A23,A31,A32,A33,
C A31Z1,A32Z2,S,A31T,A32T,Z3,Z4,Z1,Z2,A11C,A22C
dimension A11(N1,N1),A13(N1,N3),A22(N2,N2),A23(N2,N3),
C A31(N3,N1),A32(N3,N2),A33(N3,N3),A(N1+N2+N3,N1+N2+N3),
C A31Z1(N3,N3),A32Z2(N3,N3),S(N3,N3),A31T(N1,N3),Z3(N3,N1),
C A32T(N2,N3),Z4(N3,N2),Z1(N1,N3),Z2(N2,N3),
C A11C(N1,N1),A22C(N2,N2)
TRANSN='N'
TRANST='T'
N=N1+N2+N3
C let's make a copy of A11,A22,A13 and A23
do i=1,N1
do j=1,N1
A11C(i,j)=A11(i,j)
enddo
do j=1,N3
Z1(i,j)=A13(i,j)
enddo
enddo
do i=1,N2
do j=1,N2
A22C(i,j)=A22(i,j)
enddo
do j=1,N3
Z2(i,j)=A23(i,j)
enddo
enddo
c let's find the value of Z1=(A11-1)*A13. We know that A11*Z1=A13
c let's get the LU decomposition of A11
call DGETRF( N1, N1, A11C, N1, IPIVZ1, INFO)
c let's solve the system A11*Z1=A13 to find Z1.
call DGETRS(TRANSN,N1,N3,A11C,N1,IPIVZ1,Z1,N1,INFO)
c=====
c let's find the value of Z2=(A22-1)*A23. We know that A22*Z2=A23
c let's get the LU decomposition of A11
call DGETRF( N2, N2, A22C, N2, IPIVZ2, INFO)
c let's solve the system A22*Z2=A23 to find Z2.
call DGETRS(TRANSN,N2,N3,A22C,N2,IPIVZ2,Z2,N2,INFO)
C=====
c we want to fins S, That's why we have to make some products:
c A31Z1=A31*Z1
call prod(N3,N1,N3,A31,Z1,A31Z1)
c A32Z2=A32*Z2
call prod(N3,N2,N3,A32,Z2,A32Z2)
c and now we can calculate S as:
do i=1,N3
do j=1,N3
S(i,j)=A33(i,j)-A31Z1(i,j)-A32Z2(i,j)
enddo
enddo
C===============================
C we need also to find the products: Z3=A31*(A11-1) and Z4=A32*(A22-1)
c we know that Z3*A11=A31===>(A11)'*(Z3)'=(A31)'
c the first thing we have to do is transposing A31:
call transp(N3,N1,A31,A31T)
c and we solve the system making the LU decomposition before.
call DGETRS(TRANST,N1,N3,A11C,N1,IPIVZ1,A31T,N1,INFO)
c now, A31T is Z3 trasposed. Let's find Z3:
call transp(N1,N3,A31T,Z3)
c====================
c let's do the same with Z4
call transp(N3,N2,A32,A32T)
c and we solve the system making the LU decomposition before.
call DGETRS(TRANST,N2,N3,A22C,N2,IPIVZ2,A32T,N2,INFO)
c now, A32T is Z4 trasposed. Let's find Z4:
call transp(N2,N3,A32T,Z4)
C we have all the elements of the LU decomposition.
end

C===========================================

subroutine solve(A11,A13,A22,A23,A31,A32,A33,
C N1,N2,N3,S,Z3,Z4,B1,B2,B3,A11C,A22C,IPIVZ1,IPIVZ2)
integer INFO,i,j,k
CHARACTER(1) TRANSN
integer IPIVZ3(N3),IPIVZ1(N1),IPIVZ2(N2)
double precision A11,A13,A22,A23,A31,A32,A33,A,A11C,A22C
C A31Z1,A32Z2,S,A31T,A32T,Z3,Z4,Z4B2,Z3B1,B1,B2,B3,A13X3,
C A23X3
dimension A11(N1,N1),A13(N1,N3),A22(N2,N2),A23(N2,N3),
C A31(N3,N1),A32(N3,N2),A33(N3,N3),A(N1+N2+N3,N1+N2+N3),
C A31Z1(N3,N3),A32Z2(N3,N3),S(N3,N3),Z3(N3,N1),Z4(N3,N2),
C Z4B2(N3),Z3B1(N3),B1(N1),B2(N2),B3(N3),A13X3(N1),A23X3(N2),
c A11C(N1,N1),A22C(N2,N2)
TRANSN='N'
N=N1+N2+N3
c let's find the solutions:
c we know that S*X3=B3-Z3*B1-Z4*B2
c we have to do the LU decomposition of S
c then we have to solve the system. Be compute B3=B3-Z3*B1-Z4*B2 and
c after calling DGETRS, B3 will be the solution X3.
call prod(N3,N1,1,Z3,B1,Z3B1)
call prod(N3,N2,1,Z4,B2,Z4B2)
do i=1,N3
B3(i)=B3(i)-Z3B1(i)-Z4B2(i)
enddo
call DGETRF(N3,N3,S,N3,IPIVZ3,INFO)
call DGETRS(TRANSN,N3,1,S,N3,IPIVZ3,B3,N3,INFO)
c now, we have the solutions of X3 in B3
c to calculate x1, we can compute: A11*X1=B1-A13*X3
call prod(N1,N3,1,A13,B3,A13X3)
do i=1,N1
B1(i)=B1(i)-A13X3(i)
enddo
call DGETRS(TRANSN,N1,1,A11C,N1,IPIVZ1,B1,N1,INFO)
c now, the solution X1 is in B1
c to calculate x2, we can copute: A22*X2=B2-A23*X3
call prod(N2,N3,1,A23,B3,A23X3)
do i=1,N2
B2(i)=B2(i)-A23X3(i)
enddo
call DGETRS(TRANSN,N2,1,A22C,N2,IPIVZ2,B2,N2,INFO)
c now, the solution X2 is in B2
end
C======================
C======================
SUBROUTINE prod(M1,M2,M3,A,B,C)
double precision A,B,C
integer M1,M2,M3
dimension A(M1,M2),B(M2,M3),C(M1,M3)
c let's comput C=A*B
do i=1,M1
do j=1,M3
C(i,j)=0
do k=1,M2
C(i,j)=C(i,j)+A(i,k)*B(k,j)
enddo
enddo
enddo
end
C======================
C======================
SUBROUTINE transp(M1,M2,A,B)
double precision A,B
integer M1,M2
dimension A(M1,M2),B(M2,M1)
c let's transpose A and put the values in B
do i=1,M2
do j=1,M1
B(i,j)=A(j,i)
enddo
enddo
end



Saturday, October 30, 2010

SAS - Efficiency

Some SAS recommendations for making the code more efficient.
  • Avoid necessary DATA steps.
  • IF statements before loading raw data.
  • Use of WHERE statements in procedures.
  • Drop unnecessary variables by using KEEP/DROP.
  • Optimize size of variables using LENGTH.
  • Use of IF-THEN-ELSE statements when possible instead of an IF-THEN sequence.
  • Order of conditions inside the statements. The most frequent case should be the first.
  • Multiple OR operators instead of IN.
  • Use of DATA _NULL_ when creating Reports.
  • Use of permanent data sets in libraries.
  • use of PROC DATASETS to modify variables, instead of DATA statements.
  • PROC APPEND to join similar datasets.
  • Use RETAIN statement to initialize contents
  • Avoid unnecessary sorts: two-level sort instead of one-level sort and a two-level sort.
  • Sort only what we have to sort
  • Sort: Using NOEQUALS if we do not neet the relative order.

SAS - Macros

Macros are powerful tools used in SAS code which allow the execution of generic code applied to specific tables.
  • Macro variables
&name; /* Macro variable */
%name; /* Use of created macros or system macros */

Example:

%let city = "Barcelona";
TITLE "City: &city";

  • Generation of code by using macros
%macro b;
a
%mend;

proc print data=%b;
run;

SAS - Retain and Lag

In SAS, working inside one observation is easy.
On the contrary, working across observations is complicated.

In this post we will put the basics of two statements that allow to communicate information across registers.
  • RETAIN
With RETAIN, we can save the value of a variable across registers. In the following example, each register will have an additional subject field, with increasing value starting from 0.

data better;
retain subject 0;
subject = subject + 1;
input score 1 score 2;
datalines;
run;


RETAIN presents some problems, specially when dealing with missing values. If a value is missing, RETAIN does not maintain the previous non-missing value.

  • LAG
The LAG function returns the value of its argument the last time it was executed. For example,

if subj ne lag(subj) then old = new;

Here, old only changes when there is a new subject.

SAS - Basic statements

SAS is a powerful statistical language which consists bascally of an extended SQL language and other statements. In this post we will summarize the basic ones.

In this page, from the UCLA, there is a basic SAS tutorial.

  • DATA statement
data NAME_DATA;
input VAR_1 VAR_2 VAR_3 $ VAR_4; /* $ indicates VAR_3 is alphanumeric */
cards;
/* start of data reading */
123 1 foo 234
124 2 foo2 221;
run;

  • SET statement (inside DATA)
data NAME_DATA;
set NAME_DATA_2;
run; /* NAME_DATA := NAME_DATA_2 */


  • SET statement with conditional (inside DATA)
data NAME_DATA;
set NAME_DATA_2 NAME_DATA_3;
/* concatenates name_data_2 and name_data_3 */
if VAR_1 = 'yes' then answer = 1;
else answer = 0;
/* NAME_DATA := NAME_DATA_2 U NAME_DATA_3 with the additional answer field in each row */
run;

  • PROC SQL
proc sql;
/* sql code like */
create table NAME_DATA as
select * from ANOTHER_TABLE group by VAR_1;
quit;


It is important to notice that the proc sql statement ends with quit, whereas all other instructions finish with run.

  • LEFT JOIN (inside PROC SQL)
LEFT JOIN instruction allows us to join two tables by some relation between them, BUT with the additional functionality that all rows of the first table (the left table) remain at least with one observation in the resulting table, even if there is no matching between rows of the two tables.

  • SELECT DISTINCT (inside SQL code)
SELECT DISTINCT allows you to ensure that the multiplicity of each element inside a table is one, by including its identifying fields into the select distinct command.

  • How to sort a table?
proc sort data = NAME_DATA;
by VAR_1, descending VAR_2;
quit;

proc sql;
select * from NAME_DATA order by VAR_1, VAR_2 desc;
quit;


Or using the assistent. Menu Data -> Query

  • Variable names: not special chars, only '_'. Name literals finish with n, like 'Hello World!'n

Tuesday, October 19, 2010

LU decomposition with Lapack

2 pages where we can find information:

http://www.physics.orst.edu/~rubin/nacphy/lapack/compile.html

http://www.physics.orst.edu/~rubin/nacphy/lapack/codes/linear-f.html

Saturday, October 16, 2010

Gnuplot

Useful comands to plot histograms:

gnuplot> set style data histograms
gnuplot> plot "random1.dat" using 2:xticlabels(1)
gnuplot> set boxwidth 0.9 relative
gnuplot> set style histogram clustered gap 0
gnuplot> set style fill solid

Some interesting examples:
http://objectmix.com/graphics/140008-gnuplot-histogram-has-large-spaces.html

If we don't want to show the key:

gnuplot>unset key

Sunday, September 5, 2010

R - Time Series

Time Series analysis is an important part of statistics.
Bibliography
  • Time Series Analysis and Its Applications: With R Examples, by Robert H. Shumway and David S. Stoffer. In this page http://www.stat.pitt.edu/stoffer/tsa2/index.html there is the source code of the book, which is useful even if we have not read the book.
  • Introduction to Time Series and Forecasting, by Peter J. Brockwell and Richard A. Davis.
R webpage:

Sunday, August 29, 2010

R - Vectors, Lists, and DataFrames

  • A vector can be thought as contiguous cells containing elemental data.
Creation of vectors: x <- c(1,2,3)
Indexing: x[i], x[i,j]
Concatenating vectors x <- c(x,x)

  • A list (generic vector) has elements, each of which can contain any type of R object.
Creation of lists: x <- list(name_1=object_1, ..., name_m=object_m)
Indexing: x[[i]], x[[i,j]], x$a, x$"a", x[["a"]]
Concatenating lists: list.ABC <- c(list.A, list.B, list.C)
  • A data frame is a list with class "data.frame". There are restrictions:
    • The components must be vectors (numeric, character, or logical), factors, numeric matrices, lists, or other data frames.
    • Matrices, lists, and data frames provide as many variables to the new data frame as they have columns, elements, or variables, respectively.
    • Numeric vectors, logicals and factors are included as is, and character vectors are coerced to be factors, whose levels are the unique values appearing in the vector.
    • Vector structures appearing as variables of the data frame must all have the same length, and matrix structures must all have the same row size.
Creation of dataframes: accountants <- data.frame(home=statef, loot=incomes, shot=incomef)
Attach and dettach: (For lists in general). A useful facility would be somehow to make the components of a list or data frame temporarily visible as variables under their component name, without the need to quote the list name explicitly each time.The attach() function takes a `database' such as a list or data frame as its argument.

Thursday, August 26, 2010

Django - Deploy an application to alwaysdata.com (French web site)

www.alwaysdata.com is a French web site which allows free django hosting (and others) for websites up to 10Mb.
The site includes a fantastic but in French documentation which explains step by step how to publish a django site:
http://wiki.alwaysdata.com/wiki/D%C3%A9ployer_une_application_Django

Remarks:
  • ssh and ftp connections are possible.
  • There is an admin site https://admin.alwaysdata.com/ from where we need to enable the ssh user before using it.
  • The database must be created from the admin site.
  • When we make changes we may need to restart the fcgi process. This must be done from the admin web site, with the processes option, by killing the process. Immediately it is restarted.
  • Notice, as it is explained in the instructions, that we create a public directory. Some content, specially the static files, must be in that directory to be served, although the development server does not use them, it uses the files in the typical directory.
  • Important! Don't forget to redirect all queries to the main domain to the specific folder /appname/public where the project relies. This must be done from the admin panel.
  • Instructions for working with the development server are in http://wiki.alwaysdata.com/wiki/Utiliser_le_serveur_de_d%C3%A9veloppement_int%C3%A9gr%C3%A9_de_Django

HTML: How do we include one HTML file into another?

Here it is explained:
http://www.boutell.com/newfaq/creating/include.html

It depends on the way we do it: server-side, client-side, etc.
The way I used was by client side (I needed to add the page to a blogger page).
The code I used worked fine!

Saturday, August 21, 2010

R - Fitting data using Linear Models

Linear Model explanation
http://stat.ethz.ch/R-manual/R-devel/library/stats/html/lm.html

Formula object explanation
http://stat.ethz.ch/R-manual/R-devel/library/stats/html/formula.html

Example

x <- seq(0, 10, 0.1)

y <- 1 + x + cos(x) + rnorm(x)
yy <- 1 + x + cos(x)

plot(x, y, lwd = 3)
lines(x, yy, lwd = 3)

# 1 (default) includes intercept - 0 no
fit <- lm(y ~ 1 + x + I(x^2) + I(cos(x)))
fit
summary(fit)

newdf <- data.frame(x <- seq(11,2000,0.1))
newdf <- data.frame(newdf,y <- 1 + newdf$x + cos(newdf$x) + rnorm(newdf$x))
newp <- predict(fit,newdf)

plot(newdf$x,newdf$y,lwd=3)
lines(newdf$x,newp,lwd=3)

Wednesday, August 18, 2010

Django - Deploy an application to djangoeurope.com

I followed these steps to deploy a Django application into djangoeurope.com:
  • Register to djangoeurope.com and enter to the Panel (panel.djangoeurope.com)
  • Click into the installdjango option and create a project (it is called application in the webpage).
  • Run python manage.py startapp appname (the application should be called equal to our project's application).
  • Access with ssh and sftp to our space and put our original application into the corresponding folder. Overwrite all files except settings.py and settings_development.py.
  • Merge the files settings.py from djangoeurope.com and the one of our application.
  • Copy the content of settings.py to settings_development.py.
  • Run python manage.py syncdb.
  • Modify file lighttpd/django.conf to be correct.
  • Syncronize with init/appname restart, init/lightppd reload and init/lightppd restart.
Useful utilities when working are the commands
zip -r file.zip directory
,
which compresses the directory into the file, and
unzip file.zip,
which uncompresses the directory.
Caution: if we are in folder a/ and we execute unzip b/file.zip, the content of file.zip will be extracted in our working directory a/!

Sunday, August 15, 2010

How to find the package where a file is?

You can search for an ubuntu package containing a specific file (such as "algorithm.sty") by going to http://packages.ubuntu.com/. Scroll down to where it says "Search the contents of packages".

Wednesday, August 11, 2010

Django - Utilities for a beginner

In this post I recopile useful information I have found when looking for help.

  • How to specify a multiple field primary key? It is complicated, but what we can do is to specify a set of fields to be unique in the meta class of our model.
http://stackoverflow.com/questions/1624257/django-or-similar-for-composite-primary-keys
http://docs.djangoproject.com/en/dev/ref/models/options/#unique-together
  • Dynamic Django Queries
http://www.nomadjourney.com/2009/04/dynamic-django-queries-with-kwargs/

  • How can I make request.session parameters visible to a template without passing them as a parameter?
http://stackoverflow.com/questions/335231/in-django-is-it-possible-to-access-the-current-user-session-from-within-a-custom

  • A good way to customize templates for visualizing forms and form errors:
http://mikepk.com/2010/08/python_django_forms_errors_fieldsets/

  • A free online book which explains Django:
http://www.djangobook.com/

  • How to navigate in a template in a one-to-many relation:
http://solutions.treypiepmeier.com/2009/01/21/one-to-many-or-many-to-many-relationships-in-django-templates/

  • A tutorial to build a first application in Django, in four steps:
http://docs.djangoproject.com/en/dev/intro/tutorial01/

  • Query sets in Django. How can we use a field of a related object inside a filter? With object1.filter(obect2__field="aaa")
http://stackoverflow.com/questions/1086341/help-with-django-querysets

  • A person who gave up Django explains its reasons:
http://groups.google.com/group/django-users/browse_thread/thread/394701c83497e405?pli=1

  • The Catalan Django group:
http://groups.google.com/group/django-cat

  • A Django users Google Groups group
http://groups.google.com/group/django-users

  • A comparison of CMSApplications for Django
http://code.djangoproject.com/wiki/CMSAppsComparison

  • How to pass initial values to a form without validating it?
http://stackoverflow.com/questions/1882616/pass-an-initial-value-to-a-django-form-field

The general explanation of form fields is here:
http://docs.djangoproject.com/en/dev/ref/forms/fields/

  • Where can we host our Django application? A list of the most popular webs and its prices:
http://code.djangoproject.com/wiki/DjangoFriendlyWebHosts

  • Using reverse() in HttpResponseRedirect to avoid having to hardcode a URL in the view function:
http://adil.2scomplement.com/2009/01/django-using-reverse-instead-of-hardcoded-urls/

  • Changing the Model Choice Field Queryset:
from django import newforms as forms
from django.contrib.auth.models import User

class ComplaintForm(forms.Form):
user = forms.ModelChoiceField(queryset=User.objects.none())
message = forms.CharField(widget=forms.Textarea())

def __init__(self, *args, **kwargs):
super(ComplaintForm, self).__init__(*args, **kwargs)
self.fields["user"].queryset = User.objects.filter(is_staff=False)

http://oebfare.com/blog/2008/feb/23/changing-modelchoicefield-queryset/

  • Official documentation about generic views
http://docs.djangoproject.com/en/1.2/ref/generic-views/

Saturday, August 7, 2010

Django - Deploy an application

Once we have a Django application finished, we probably want to put it in a "real" web server.

This page explains how we can do it, using Apache and mod_wsgi, a module that allows Apache to support any Python application which supports the Python WSGI interface, like Django. We briefly summarize the steps in this post.
sudo aptitude install libapache2-mod-wsgi
  • We can test it by creating a wsgi script and modifying the Apache configuration file httpd.conf to allow the access to this file. For example we can make an script called /usr/var/www/scripts/myapp.wsgi
def application(environ, start_response):
status = '200 OK'
output = 'Hello World!'

response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
start_response(status, response_headers)

return [output]


and adding some commands to the httpd.conf file, found in Ubuntu inside /etc/apache2 folder.
  • To deploy our Django application, we must create a file called django.wsig inside a folder, like mysite/apache, with the following content:
import os
import sys

sys.path.append("django_folder_application_path")
os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings'

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

and the httpd.conf file must have the content explained in this page.

  • We must take into accout several things:
    • Make sure the folders have the correct permissions. Specifically, they must have read and execute permissions for the www-data user, the apache server user.
    • If we are using sqlite databases, the database file and its directory must have writing permissions, as it is explained here. Also we must ensure that the db name contains the full path.

Thursday, July 29, 2010

Com contolar processos que funcionen en mode text - Screen

Screen és un programa que permet (entre altres coses) controlar processos que funcionen en terminals remotament.

Funciona de la següent manera:

- Obrim una finestra, (que controlarem remotament). Per a fer-ho, des d'un terminal executem:
screen

- Obrim altres finestres des d'aquesta, des d'on podem cridar nous processos
Ctrl A + c (de create)

- Per a desacoblar les finesters
Ctrl A + d (de dettach)

Ara el terminal no conté els processos, i obrint-ne un altre els podem recuperar:
screen -r (de recover)

Per a més info: http://www.kuro5hin.org/story/2004/3/9/16838/14935

Thursday, July 22, 2010

Codificació de caràcters UTF-8 / ISO-8859-1

La codificació de caràcters pot portar molts problemes.
Aquí explica com s'ha de fer per convertir tot el sistema a UTF-8

http://www.oreillynet.com/onlamp/blog/2006/01/turning_mysql_data_in_latin1_t.html

Python - Profiling

Per a fer profiling amb python:

python -m cProfile -o cProfile_stats prog_python.py

El resum s'escriu al fitxer cProfile_stats. Per a llegir-lo:

python
>>> import pstats
>>> p = pstats.Stats('cProfile_stats')
>>> p.sort_stats('time').print_stats()

Més informació aquí: http://docs.python.org/library/profile.html

Wednesday, July 21, 2010

R - Problema al carregar una llibreria

M'he trobat amb el següent problema
  • Amb un usuari vaig instal·lar una llibreria.
  • Des d'un altre usuari (l'usuari apache www-data) volia carregar la llibreria però no podia.
  • Accedint com a super usuari em carregava la llibreria correctament.
Explicació:
  • Hi ha dos tipus de repositoris on s'instal·len les llibreries: als directoris locals dels usuaris i en global per a tots
  • Per defecte s'instal·len les llibraries als directoris locals dels usuaris
  • Només quan l'usuari no té /home és quan les llibreries s'instal·len en global.
  • Per a canviar d'usuari, si no en sabem el password, podem accedir primer al super usuari i després canviar a l'usuari en que volem: sudo su / su usuari2

Tuesday, June 29, 2010

WEKA i MySQL

Com connectar WEKA amb MySQL.

Pàgina on ho explica: http://estudiandobi.blogspot.com/2008/02/conectar-weka-con-mysql.html

Coses a tenir en compte:
- Al directori home/username hi ha d'haver el fitxer DatabaseUtils.props
- S'ha de carregar la llibreria mysql-connectors-java.jar al CLASSPATH

A la web http://mrtextminer.wordpress.com/2008/03/16/setup-weka-and-libsvm-for-weka-in-debian-etch/ hi ha informació de com connectar WEKA amb la llibreria de SVM.

Sunday, June 27, 2010

Linux - Apache - PHP - MySQL

Resum de l'estructura dels directoris en un Debian/Ubuntu

- /etc/init.d conté els progrmes que s'executen, com ara apache2 o mysql
- /etc/mysql conté la configuració de mysql
- /etc/apache2 conté la configuració de apache2
- /etc/phpmyadmin conté la configuració de phpmyadmin
Per poder veure phpmyadmin a localhost/phpmyadmin s'ha de modificar el fitxer /etc/apache2/apache.conf afegint-li Inlclude /etc/phpmyadmin/apache2.conf

Wednesday, June 23, 2010

MySQL - Query Cache

La Query Cache emmagatzema consultes prèviament realitzades per així poder respondre-les més ràpidament.
  • És key-sensitive
Si deixem una consulta a mig fer, llavors la Query Cache d'aquesta consulta queda en un estat dubtós, i quan tornem a fer la consulta no retorna res, es queda bloquejada.

Per a evitar això podem fer un
RESET QUERY CACHE

Més info:
http://www.mysqlfaqs.net/mysql-faqs/Speed-Up-Queries/What-is-query-cache-in-MySQL
http://dev.mysql.com/doc/refman/5.0/en/query-cache-configuration.html

Saturday, June 12, 2010

MySQL - Com connectar amb una base de dades remota usant CPANEL

Aqui esta explicat:
http://paragonhost.wordpress.com/2007/03/10/remote-mysql-connection-to-a-cpanel-server-how-to/

Nota.
Quan engeguem l'ordinador, per defecte el servidor mysql no està inicialitzat. Per tal de fer-ho, hem d'accedir al directori del disc extern on hi ha la base de dades (ex. media/disk/mysql) i des d'allà engegar el servidor mysql (sudo /etc/init.d/mysql start).

Friday, June 11, 2010

MySQL - Com canviar el directori on guardar les dades

Extret de http://www.freelancis.net/ressources/mysql/changing_directory

El directori on s'emmagatzemen les bases de dades de mysql per defecte és:
/var/lib/mysql

Si volem emmagatzemar la base de dades a un disc extern, per exemple, hem de fer el següent:
  • Parar el dimoni mysql
/etc/init.d/mysql stop
  • Crear el directori arrel on vulguem i donar-li permisos per accedir des de la BD
mkdir /home/mysql
chown -R mysql /home/mysql
chgrp -R mysql /home/mysql
  • Copiar les dades bàsiques, usuaris, etc
cp -Rp /var/lib/mysql/* /home/mysql/

  • Canviar la configuració MySQL
vi /etc/mysql/my.cnf

Canviar la línia datadir = /var/lib/mysql per datadir = /home/mysql
  • Activar de nou el dimoni MySQL i provar que va bé creant una nova BD
/etc/init.d/mysql start

  • Alliberar espai. Tots els fitxers del directori /var/lib/mysql, excepte
debian-5.0.flag
mysql_upgrade_info
i el directori mysql


poden ser eliminats

MySQL - Varis

El fitxer /etc/mysql/debian.cnf conté el nom d'usuari (debian-sys-maint) i la contrassenya de l'usuari principal de mysql.
  • Accedir a mysql:
/etc/init.d/mysql -u debian-sys-maint -p
  • Treballar amb la base de dades correcta, la que gestiona els usuaris:
mysql> use mysql;
  • Canviar contrassenya d'un usuari (per exemple, el root si no la sabem):
mysql> update user set password=PASSWORD("NEWPASSWORD") where User='vivek';
  • Crear un nou usuari amb privilegis:
mysql> CREATE USER 'miquel'@'localhost' IDENTIFIED BY 'passmiquel';

mysql> GRANT ALL PRIVILEGES ON *.* TO 'miquel'@'localhost' WITH GRANT OPTION;Query OK, 0 rows affected (0.00 sec)

  • Actualitzar privilegis:
mysql> flush privileges;
mysql> quit

PHP - Com veure els errors de PHP?

A la "versió de desplegament", és a dir, als servidors que funcionen de veritat, els errors de PHP estan ocultats. Per tal de veure'ls jem de modificar el següent:

Fitxer
/etc/php5/apache2/php.ini
Linia 531
display_errors = On/Off

Això ho he necessitat per a poder veure els errors del servidor Apache instal·lat per defecte a l'ordinador.

Linux - Servidor - Carpetes

Engegar / Aturar el servidor Apache

/etc/init.d/apache2 start | stop | restart

Directori on afegir els fitxers

/var/www

Si no volem moure els fitxers de dins un directori a aquest directori, podem fer un softlink de la següent manera

ln -s /path/OrigDir /var/www/newDir

R - Com fer un gràfic senzill

# Resultat en un fitxer pdf. width=7, height=7 by default. (in inches)
pdf(file="prova.pdf",width=7,height=7)

# Parametres dels grafics. mfrow es per a posar mes d'un grafic a la figura. Cada grafic comença amb plot()

# Els altres son per als marges
#par(mfrow=c(3,3), mar=c(3.5,3.5,1.5,0.5), oma=c(0,1,1,1))

a <- matrix(ncol=2,nrow=10) a[,1] = c(1,2,3,4,5,6,7,8,9,10) a[,2] = c(3,5,3,87,9,63,12,51,11,22) b <- matrix(ncol=2,nrow=10) b[,1] = c(1,2,3,4,5,6,7,8,9,10) b[,2] = c(13,15,23,17,59,3,27,75,18,32) plot(a,col="green",pch=17,ylim=c(-1,100),xlab="alpha",ylab="Variation of Information",main="sites.net"); lines(a,col="green") points(b,col="red",pch=20); lines(b,col="red") legend("topright",c("a","b"),col=c("green", "red"),pch=c(17,20)); # Tanquem el fitxer pdf dev.off()


El resultat és

R - Utilitats

El llenguatge R és un llenguatge matemàtic molt útil. En destaco un parell de funcionalitats:
  • La funció match(), que retorna els índexos dels elements buscats.
  • par(mfrow=c(3,3)), que permet mostrar 3x3 gràfics en una sola finestra.
  • pdf(...,width=7,length=7), que permet imprimir a un fitxer pdf els gràfics. Acaba amd dev.off()
  • mca, que fa un anàlisi de correspondències múltiples. Dins la library MASS
  • hclust, que fa un hierarchical clustering
  • hist2d, image, image.plot, que són eines gràfiques
  • choose(n,m) calcula combinacions de n elements en grups de n
  • ginv(M) calcula la inversa de M. Dins la library MASS
  • binary(i,d) obté la representació en binari de d digits del nombre i. Dins la library wle. binary(i,d)$binary és un vector de zeros i uns.
  • write.table(b,file="filedest", row.names=FALSE, col.names=FALSE)
  • plot(a,col="green",pch=17,ylim=c(-1,100),xlab="alpha", ylab="Information", main="net")
  • Creacio de funcions: nomfunc <- function(parameters){... retValue}

Django - Basics

Updated 2010-08-07

Django is a Web framework that allows making webs in a fast and easy way. As his lemma says, it is a framework "for perfectionists with deadlines".

Here there is a basic tutorial, divided in four steps, which explains its fundamentals.
  • Step 1. Creating the data Model.
  • Step 2. Activating the automatic web Administration.
  • Step 3. Creating the public interfaces - Views.
  • Step 4. Form processing and depurating the code.
Django structures the code in three parts, following the classic Model-View-Controller scheme, although it is slightly different. Its parts are the following:
  • Model, which contains the relational scheme of the database.
  • Views, which deploy the web functionalities.
  • Templates, which concrete the final HTML code which is seen in the web application.
The basic files and folders in which a Django application is divided are the following:
  • settings.py File with configuration parameters
  • manage.py Application which allows to run the development server, to reset the database and other functionalities.
  • urls.py Module which works as an URL dispatcher.
  • application/models.py Contains the Model of the application.
  • application/views.py Contains the Views of the application.
  • application/forms.py Contains the forms used in Views and Templates.
  • application/admin.py Admin default module.
  • templates/ Contains different Templates.
  • files_media/ Contains static files like images, CSS or JavaScript files
Some typical problems when working with Django:

Modification of the Model
  • Once the model is created, we may need to change it, changing tables, adding new ones, and so on.
  • If we allow to loose the existing data, the easiest way to do that is the following, explained in the answer 10 of the FAQ's: to delete the existing tables with the command
manage.py reset appname
and re-creating them with
manage.py syncdb
manage.py sql appname
  • If we do not want to loose the existing data we must be more careful:
    • If we only add tables without modifying or deleting other tables, the application of the re-creating commands works well.
    • If we also want to modify an existing table we must be more careful: We must modify manually the database. A good way to do that is with the alter table command of sqlite3. We must be very careful and we must maintain in each moment the integriity between the database and the defined model:
      • Fields names: The field names of the table finish with _id.
      • Fields are not null by default at the model and null by default in the database. So, it is not a problem. If we want the database to store not null fields, we must indicate it:
CREATE TABLE "myapp_informacio" (
"id" integer NOT NULL PRIMARY KEY,
"usuari_id" integer NOT NULL
REFERENCES "myapp_usuari" ("id"),
"missatge" varchar(250) NOT NULL,
"data_consulta" datetime NOT NULL
)
      • When we want a field to be not necessarily null at the database we must modify the model adding the "restrictions":
camp = models.CharField(null=True,
blank=True,max_length=5)

The Gimp: Making Colors in a GIF Transparent

From http://aplawrence.com/Linux/crousegif.html.

Sometimes when working with an image you want to make a certain color transparent. When working with a gif file this would make a round circle look round on any color background. This is actually very simple once you do it once. Finding the information for this took me a while so I thought I would pass it on to anyone that was interested.

1. Open your image in the gimp.

2. Right click the image and go to LAYERS then ADD ALPHA CHANNEL. You won't notice anything happening, but don't be concerned. It basically adds a transparent layer at the bottom of your image so when we erase the colors.....it's shows the transparent layer. Which of course would show whatever was under it on the screen.

3. Right click on the image again and go to SELECT and then down to BY COLOR. A window that is all black opens up. Don't change any of the settings....just use the defaults for now.

4. Now click on the color in the image you want to be transparent. These colors will now show up outlined.

5. Right click on the image again and go to EDIT and then down to CLEAR. This should now erase the outlined color you just picked from the image and the "transparent gimp checkerbox" should show through. This is the Gimps way of showing you that section is now transparent.

6. Right click on the image and choose SAVE AS and make sure to save as a GIF file if you want the transparency to work on the web.

Instalation of Freeling - Language Processing Tools in Ubuntu 9.10

Documentation is found in Freeling installation page. However, I've had several problems when following the described steps.

I followed the installation from .tar.gz source packages. Here I will describe the differences between what I did and the explanation from the .tar.gz installation page.

1. Install development tools

Ok

2. Install packaged requeriments
I installed the following package instead of
sudo apt-get install libboost-dev
sudo apt-get install libboost1.40-all-dev
(due to problems with libboost-filesystem)

3. Install non-packaged requeriments

Ok

4. Install Freeling
Previous installations OK
Installation of Freeling
sudo ./configure
sudo make
sudo make reported a bug when using libtool. I solved them by following the explanation of here. Basically, with the addition of these lines in file Freeling/libtool
# An echo program that does not interpret backslashes.
ECHO="echo"
+#well this $ECHO is referred to as $echo everywhere in this file. So:
+echo="$ECHO"

# Used to examine libraries when file_magic_cmd begins with "file".
MAGIC_CMD=file

sudo make
sudo make install
Problems with freeling/data/Makefile.am. I solved them by following the explanation of here.
Basically, with the Modification of all lines like
asdata_DATA = es/*
to
asdata_DATA = es/*.*
These lines refer to asdata, cadata, esdata, gadata, commondata, itdata... (one for each lang).
sudo make install
Now all was ok!

Examples
In the directory Freeling/src/main/simple_examples/
g++ -o sample sample.cc -ldb_cxx -lmorfo -lfries -lomlet -lpcre
I needed to add the library lboost_filesystem
g++ -o sample sample.cc -ldb_cxx -lmorfo -lfries -lomlet -lpcre
-lboost_filesystem

Error d'execució al carregar una llibreria

L'error era aquest: la llibreria estava posada a una ruta errònia. Poso l'explicació que m'ha servit.

liblablabla.so.5: cannot open shared object file: No such file or directory

Problem: An application is running on Linux 7.x which uses gcc 3.2.x. It gets one of the following error messages:

libstdc++.so.5: cannot open shared object file:
No such file or directory
libgcc_s.so.1: cannot open shared object file:
No such file or directory
Explanation: libstdc++.so.5 (the library for gcc 3.2.x) is installed in /usr/local/lib However, the system will only seach /usr/lib (where the 2.95.0 and other older libraries are stored).

Solution: Create symbolic links to make the libraries appear in the old folder:

ln -s /usr/local/lib/libblablabla.so.5
/usr/lib/libblablabla.so.5
ln -s /usr/local/lib/libblablabla.so.1
/usr/lib/libblablabla.so.1

Joomla - Cosetes

  • Canviar el títol per defecte de la pàgina: Anar a Menus -> Main Menu -> Home -> Edit -> Parameters (System) i canviar el títol (http://cmsjoomla.wordpress.com/2008/10/16/modificar-el-texto-bienvenidos-a-la-portada/)
  • Canviar la icona per defecte de la pàgina: Fitxer nom_plantilla/favicon.ico
  • Component joomla per reproduir mp3. Es un Mambot i es diu *1 Pixel Out Audio Player*.
*Per la versió 1.0x del joomla: *
Descàrrega: http://duvien.com/system/files/BOT_onepixelout-player-v123b.zip
Informació: http://duvien.com/onepixelout-player-ported-joomla-10x

*Per la versió 1.5 del joomla: *
Descárrega: http://duvien.com/system/files/plg_onepixelout-player-v1_5_2.zip
Informació: http://duvien.com/1pixelout-player-152-released

Per fer-lo servir ho has de fer així, posant /{audio}arxiu MP3{/audio}/

Ortografia a un document LATEX

NO admet format UTF8
ispell -t -T latin1 -d idioma tutorial-latex.tex

-t Latex format
-d catalan/spanish

FUNCIONA BÉ
aspell -t -l ca check document

Replace r paraula_nova

Com fer un GIF animat amb el GIMP?

  • Primer amb les eines ScaleTool i Rectangle de selecció podem aconseguirles imatges de la mida que volem.
  • Quan ja les tenim totes (amb extensió gif) obrim la primera.
  • Per a la resta les obrim amb la opció Open as Layers.
  • En el menú Dialogs -> Layers podem configurar l'ordre... etc.
  • Quan anem a guardar la imatge ens preguntarà si volem que sigui estatica (NO)o dinàmica (SÍ), així com el temps entre foto i foto (en milisegons).
I ja tindrem el nostre gif animat. Això sí: la resolució no és massa bona.

Per allotjar-lo a un servidor Web que permeti visualitzar l'animació: http://imageshack.us/
No cal registrar-se


Com convertir una impatge a pdf?
  • (des d'Ubuntu) Programa mogrify. Ex. mogrify -format pdf imatge.jpg

Programes informàtico-matemàtics

Llibreria per al càlcul de VAPS i VEPS molt grans: ARPACK (en Fortran), ARPACK++ (interfície per a C++)
  • Per a Ubuntu hi ha un paquet deb.

SIMPLEX - Programació lineal
  • CPLEX: El millor, de pagament
  • GLPK: Free software, va bé. Implementat en C++, té una interfície en R. Els problemes s'especifiquen en MATHPROG, un llenguatge per a especificar problemes.
  • COIN-OR: Free software

Programes matemàtics gratuïts

Software educatiu

Joomla - Passos molt bàsics per a fer una web

Menús: Agrupacions d'enllaço, imatges...
  • Es pot triar l'ordre i el nivell d'accés.

Contingut: Categories -> Seccions -> Articles
  • FrontPage

Components:
  • Remository
  • Weblinks: Categories -> Links

Extensions:
  • Module Manager: Són els que es veuen a les pàgines. Formats per menús i altres mòduls.
  • Localització (esquerra, dreta, ..., ordre)
  • Nivell d'accés (públic, privat, ...)
  • Assignació de menús: Menús i elements des dels quals serà visible el mòdul.

Eines AUTO per a la compilació de projectes

Al directori arrel /
configure.in
Makefile.am
A cada carpeta
Makefile.am
Executem
aclocal : arxiu aclocal.m4 amb macros
autoconf : configure.in -> scripts configuració
automake -a (--foreign) : Makefile.am -> Makefile.in
autoheader : Plantilla configure
Finalment
./configure
make

Accedir per ssh dins lsi

ssh nomusuari@login1.upc.edu

ssh2 nomusuari@nommaquina