Incompatibilidad entre Autofirma 1.4.x / 1.5.x y Google Chrome 58

El problema

Hace ya 1-2 semanas me ha dejado de funcionar Autofirma con Google Chrome bajo OS X.

Mirando la consola en las herramientas para desarrolladores he podido encontrar el siguiente error: net::ERR_RESPONSE_INSECURE. Al intentar conectar rápidamente con Google Chrome a https://127.0.0.1:<Puerto indicado en mensaje de error> ya que Autofirma se cierra después de un corto tiempo, y activando el panel de seguridad, obtengo finalmente​ más información: commonName matching error.

Googleando un poco, me he encontrado con este interesante thread: https://groups.google.com/a/chromium.org/forum/m/#!topic/security-dev/IGT2fLJrAeo

Resumen: Si un certificado no tiene la extensión subjectAltName mal asunto, porque se dejará de soportar el hostame contra el campo commonName. La justificación es que usar el campo commonName para decidir si el hostame solicitado se corresponde con el certificado puede ser ambiguo, ya que se emplea tanto para IPs como para nombres de dominio.

Es justo lo que pasa al certificado generado de forma automática durante la instalación de Autofirma.

Soluciones

• Usar otro navegador: Hasta ahora Firefox me sigue funcionando bien.
• Forzar el uso de servidor intermedio. Esta es la opción usada por nosotros, ya que la conexión directa entre Autofirma y el navegador nos ha dado muchos problemas ya (Antivirus, OS X 10.11).

Esperemos que pronto salga una versión nueva que resuelva el problema.

hg share: Sharing mercurial repository between different clones / checkouts

Our starting point

At our company, we developed a product based on Django. To manage code changes, we use mercurial and to manage all dependency stuff we use buildout + setuptools. buildout recipes are wonderful if you need to do other things than just pulling code and resolving and building library dependencies. These thing could be:

• Building any binary from source. We use it for building Nginx, part of product
• Generating config files. We used it for generating configuration files for nginx, supervisor, etc.
• Generating SSL Certificates
• etc.

Our deployments use a shared product base with its mercurial repo and customer specific project customizations which are held on separate mercurial repos. Managing changes with mercurial (or any other SCM system) allows us to:

• Deploy quickly any hot fixes
• Share code changes easily merging or "grafting" changesets between branches.
• Get exhaustive change history information.

The problem

When working on several projects at the same time, it's not easy to share the same "buildout" project because each one has it's own settings, customizations, and so on. That led me to have a copy for each customer. Each buildout is about 1GB.

As the number of customers rise, the space required to hold all buildouts is getting quite big.

The solution

Using shared mercurial repository

A mercurial repository can be divided into:

• a history tracking store where all changesets reside
• the state which is basically a pointer to an entry in the history
• a local copy, which hold any changes which are not commited

The store can be shared between several clones / checkouts / repositories. This is just what the mercurial share extension (hg share) does. 

The syntax is similar to the "hg clone" command: hg share <local source repo> [<dest name>]

One of the advantages is that a change is directly visible to each clone. This saves a lot of pulls. But care should be taken, because strips / rollbacks apply to them all. This could leave a repository pointing to a non-existing (anymore) state.

Using shared eggs and download-cache directories

These directories hold nearly the same info between different buildouts, so it's easy to share them. The solution I used is to simply use symbolic links to some globally shared directories. Another solution would be to specify specific eggs and download-cache directories inside buildout parameters (eg. using a "develop.cfg" invoked from "buildout.cfg" which inherits from a "base.cfg").

A + B

I worked out a little script which replaces automatically each mercurial repository with a shared one and unifies the eggs and the download-cache directories.

Applying both changes to each of my buildouts reduces them by more than 65% including the shared part of eggs and download-cache. This is quite a good saving.

Fast PDF scaling with page numbering under Ubuntu

The problem

We want a backend process to scale PDF files and number pages. Currently, wer'e using some Java code bases on the last LGPL iText version (2.1.7) which does PDF scaling and stamping. But the code includes some features for custom output formatting (text tables, barcodes) for footers and margins written in Java, so that only software developers have the knowledge to customize and recompile the code. Wouldn't it be nicer if the customer could customize these output formats directly?

What we need:

• PDF stamping feature
• Page numbering
• Page scaling

We've used PyPDF2 and xhtml2pdf in the past, but it may be too slow for big documents.

The proposed solution

pdfjam is a package with a bunch of scripts for pdf manipulation based on pdflatex / pdftext command line included in Tex Live Binaries packge. On Ubuntu, you can get it from the standard repositories.

Scaling

The following command line scales a PDF input file:

pdfjam --scale 0.9 --outfile output.pdf input.pdf

It's very quick. On my machine it takes less than 1s for a 120 page 2.1MB PDF file.

Page Numbering

With some additions we can generate page numbers. Note: the following command should be a one-liner:

pdfjam  --preamble '\usepackage{fancyhdr} \topmargin 85pt \oddsidemargin 140pt \pagestyle{fancy} \rfoot{\Large\thepage} \cfoot{} \renewcommand {\headrulewidth}{0pt} \renewcommand {\footrulewidth}{0pt} '  --pagecommand '\thispagestyle{fancy}' --scale 0.9 --outfile output.pdf input.pdf

This is still very quick. Some explanations go here:

• The preamble argument is just the text which goes into the .tex command file fed to pdlatex just before the "\begin{document}" part.
• The --pagecommand is an additional argument which goes into the "\includepdfmerge" command
• If you want to have a look into the generated .tex command file, add --no-tidy to the command line.
• The "topmargin" and "oddsidemargin" are set for A4 page size. You may experiment with your own preferences.

Page Numbering with the "{page} of {pages}" format

If we would like to write out page numbers like this, we need the lastpage Tex package. Now the pdflatex command (called from pdfjam) must be invoked twice. This requires changing the pdfjam shell script. Just replace the line:

$pdflatex $texFile > $msgFile || {

with something like this:

$pdflatex $texFile > $msgFile && if grep 'xdef' $auxFile > /dev/null ; then $pdflatex $texFile >> $msgFile ; fi || {

i.e.: If the aux file contains any xdef definition, we'll do second pass.

For Ubuntu, the lastpage Tex Live package is included in the texlive-latex-extras package. If you don't want to install the recommended documentation, you could run the following command:

sudo apt-get install --no-install-recommends texlive-latex-extra

Now, let's change the page numbering format:

pdfjam --preamble '\usepackage{fancyhdr} \usepackage{lastpage} \topmargin 85pt \oddsidemargin 140pt \pagestyle{fancy} \rfoot{\Large\thepage\ of \pageref{LastPage}} \cfoot{} '  --pagecommand '\thispagestyle{fancy}' --scale 0.9 --outfile output.pdf input.pdf

This doubles the time required to generate the document, but still 1.8s for my 120 pages document.

Escaping single quotes in bash

The problem

We want to define an alias for deleting python .pyc files. The alias definition:

alias rmpyc="find . -name '*.pyc' -delete

does not work correctly

Solution

Combining the bash rules for quoting:

• Any variable and pattern is escaped whenever it is NOT enclosed within single quotes
• Backslash escaped single quotes NOT enclosed within single quotes produce literal single quotes
• Expressions with different quotes can be directly combined, e.g.:

echo "Double quotes: "'"'' and single quotes: '"'" 

Applying both rules we can write:

1. alias rmpyc='find . -name '"'"'*-pyc'"'"' -delete'

or

2. alias rmpyc="find . -name '"'*.pyc'"' -delete"

or

3. alias rmpyc='find . -name '\''*-pyc'\'' -delete'

Detailed explanations

1. We start using single quotes because we want to define a literal. To obtain a literal single quote, we stop quoting the first literal (find . -name) and concatenate a single quote enclosed within double quotes, then we write the file pattern within single quotes to avoid its expansion. Again, we concatenate a single quote enclosed within double quotes and finally we add the rest of the string ( -delete).
2. We start with double quotes while there is no pattern to be expanded. This allows us to include literal single quote. After we ended the first string (after the first literal single quote), we concatenate the file pattern enclosed within single quotes to avoid file pattern expansion. Finally we add the rest of the string within double quotes (starting with the second literal single quote).
3. We start with single quotes like in 1. until we want a literal single quote. To get it, we end the first string (with a single quote) and then we write an escaped single quote (backslash single quote). We continue with the file pattern enclosed within single quotes to avoid its expansion. Then we add the second escaped single quote like before. Finally we concatenate the rest of the string.

Hints for Java JMX monitoring (for Tomcat, Alfresco, Liferay, etc.)

The problem

We want to monitor an Alfresco server which is not directly accesible from the outer world. It sits inside a (VMWare) private virtual network behind a firewall.

Have you ever tried to access JMX in private virtual nets behind a firewall? 

It's not easy at all, because of the way JMX connection establishment works: The client connects to a well know RMI registry host:port. If no additional variables are set, the Java VM does these things:

1. Guess it's own IP, based on the hostname and /etc/hosts.

2. Allocate dynamically a port to receive "server" connections

3. Send the data to the client, so it can do the connection.

In our scenario (not directly reachable Alfresco server in a private virtual network), this is a real nightmare.

Here are some solutions.

Solution 1: Create / use VPN

I don't have this solution at hand, so I'll jump to the next one.

Solution 2: Fix and expose JMX ports to the outer world (protected by firewall)

1. Download Apache-Tomcat's extra catalina-jmx-remote.jar for your version of Tomcat and drop it into the tomcat/lib folder

2. Add to tomcat/conf/server.xml something like this:

<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" rmiRegistryPortPlatform="8555" rmiServerPortPlatform="8556"/>

3. Add the following variables to tomcat/bin/setenv.sh (or tomcat/scripts/ctl.sh, in case of Alfresco):

CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote "

CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

CATALINA_OPTS="$CATALINA_OPTS -Djava.rmi.server.hostname=`hostname`"

4. Open your firewall for the given ports and source IPs.

Notes:

• The java.rmi.server.hostname value is sent verbatim to the client. This makes it possible that the hostname resolves at the server to one IP and at the client to another.
• We disable SSL, we're supposing that the access is protected  through firewall.
• We suppose that the firewall maps the ports for the outer world to the server in out private network
• In most articles we can find in Internet,  java.rmi.server.hostname should be a valid IP, but this is just another reason why it's so difficult to get the right configuration. I inspected network packets with ngrep and found that the value is send verbatim.
• At the server, the hostname value should resolve locally. When I used some outer IPs, Tomcat didn't start up correctly (taking a long long time...)

Solution 3: Use Jolokia and expose some special URLs to the outer world

Jolokia is an agent which translates JMX queries and operations to REST-HTTP/JSON. It's really easy to write a Nagios check script. I did one in Python with took something like an hour.

What I did:

1. Download the WAR from the Jolokia download page.

2. Unzip the WAR to edit web.xml

3. Modified the web.xml, uncommenting the authentication things

4. Zip the war again and drop it into the tomcat/webapps folder

5. Add a user with the "Jolokia" role to conf/tomcat-users.xml

6. Restart Tomcat

7. Test it with a browser at /jolokia/ (The browser should show an authentication dialog.)

8. Search for jolokia nagios plugins or write one.

With a little bit more of time, I modified my Nagios plugin (which I use from Shinken, not Nagios) to display all Heap Memory data into MBytes or percentage, so you can something like this:

./check_jolokia_heap -U http://......  -c 80% -w 90% -u -p

and here is an example output (should be in one line):

JMX OK HeapMemoryUsage.used=439.57{max=1185.5;init=1248.0;used=439.57;committed=1185.5}|HeapMemoryUsage.used=439.57;998.4;1123.2

Note that, although we specify -w and -c arguments in percentage the values are translated into MBytes.

If the -P flag is given the values are translated into percentages:

JMX OK HeapMemoryUsage.used=34.96%{max=1185.5;init=1248.0;used=436.32;committed=1185.5}|HeapMemoryUsage.used=34.96%;80.0%;90.0%

If you're interested, leave a comment.

Solution 4: Invoke a JMX monitoring through SSH

Before we begin, let's talk about the pros and cons:

• Pros: You don't have to hassle with JMX configurations.
• Cons:
• The JMX monitoring command is invoked at the target machine. Make sure you have enough memory
• If there is any SSH issue, the command will fail, although the JVM may work correctly
• You need SSH, of course.

Basically, you don't have to bother about JMX ports, firewalls, etc. Just install the monitoring plugin in the target machine and invoke it through SSH.

Now, the question is, how to to invoke it automatically with no direct SSH connection? (Remember that the host is not directly accesible?)

Here you have two solutions:

a) Configure you firewall to forward SSH port to the target machine

b) Use SSH ProxyCommand: Define in the ~/.ssh/config SSH configuration of the monitoring account something like this:

# Our proxied destination host

Host destination-host

  ProxyCommand ssh intermediate-host -W %h:%p

Make sure you can reach the intermediate host without password authentication:

ssh-keygen   #if you don't have already any keypair generated
ssh-copy-id intermediate-host

Now, test you connection to the destination-host:

ssh destination-host

You should get a prompt if you trust the destination certificate's fingerprint and after that the password prompt. If everything works as expected, just copy your public key to the destination host:

ssh-copy-id destination-host

Finally, copy your monitoring plugin to the destination host and invoke it, e.g. in Nagios / Shinken, your command definition could be something like this

define command {
    command_name    check_tomcat_mem_heap
    command_line    $NAGIOSPLUGINSDIR$/check_jmx \
        -U service:jmx:rmi:///jndi/rmi://'$HOSTADDRESS$':'$ARG1$'/jmxrmi \
        -O java.lang:type=Memory -A HeapMemoryUsage \
        -K used -w '$ARG2$' -c '$ARG3$
}

Liferay 6 and Sentry

Introduction

Sentry [1] is a great tool for error tracking and Liferay [2] is a very popular portal software that we deploy for out customer as part of our main product.

Log4j configuration with Liferay

As stated in [3], custom log4j configuration is done adding these files:

• portal-log4j-ext.xml
• log4j.dtd

to folder: tomcat-[version]/webapps/ROOT/WEB-INF/classes/META-INF where [version] is something like 7.0.23 and depends on the concrete Liferay version you are working with.

The portal-log4j-ext.xml overrides the file portal-log4j.xml which can be found inside portal-impl.jar (in webapps/ROOT/WEB-INF/lib). You can get a copy from here [4]. The companion file log4j.dtd can be found here [5].

Get Sentry Java Client (raven-java)

To log any errors in our Liferay instance to Sentry, we need a Sentry Java Client which works together with log4j (a Log4j appender) and can be downloaded from here [7].

Although Liferay already includes log4j 1.2.x we choose the jar that includes all dependencies (for a reason I explain below):

Download the file raven-log4j-[version]-jar-with-dependencies.jar where [version] is currently: 1.0-SNAPSHOT. 

Configure Liferay to log to Sentry

This is done it two steps:

1. Copy the downloaded raven-log4j JAR to tomcat-/lib
2. Customize portal-log4j-ext.xml

Copy raven-log4j file

Copy the downloaded raven-[version]-jar-with-dependencies.jar to tomcat-[version]/lib.
Although all the java dependencies should normally live together with the web application and the downloaded file contains log4j 1.2.x, we have chosen this approach for the following reasons:

• Then specific log4j appender and Sentry java client (raven-java) will be available for other deployed web applications. Take into account that in Liferay deployment each extra portlet is considered a separate web application with its own J2EE application context. Each of them will have to be configured individually to log to Sentry.
• The log4j version included in raven-log4j-[version]-jar-with-dependencies.jar has the same major and minor version (1.2), i.e. the Sentry log appender (SentryAppender and AsyncSentryAppender) should be 100% compatible. Special caution should be given when using other versions.
• The tomcat/lib contains class libraries (JAR) which are shared between all of the deployed web applications (WAR). But (!) the J2EE classloader magic avoids calls from a shared class library back to a web application class. In this case, it doesn't happen.
• The web application classes and class libraries have preference when searching for a class

Customize portal-log4j-ext.xml

Following this example [6] where Nuxeo is configured for Sentry, we have to add an appender and activate it in portal-log4j-ext.xml, like this:

 <?xml version="1.0"?>  
 <!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">  
 <log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">  
   <appender class="org.apache.log4j.ConsoleAppender" name="CONSOLE">  
     <layout class="org.apache.log4j.PatternLayout">  
       <param name="ConversionPattern" value="%d{ABSOLUTE} %-5p [%c{1}:%L] %m%n" />  
     </layout>  
   </appender>  
   <appender class="org.apache.log4j.rolling.RollingFileAppender" name="FILE">  
     <rollingpolicy class="org.apache.log4j.rolling.TimeBasedRollingPolicy">  
       <param name="FileNamePattern" value="@liferay.home@/logs/liferay.%d{yyyy-MM-dd}.log" />  
     </rollingpolicy>  
     <layout class="org.apache.log4j.PatternLayout">  
       <param name="ConversionPattern" value="%d{ABSOLUTE} %-5p [%c{1}:%L] %m%n" />  
     </layout>  
   </appender>  
   <appender class="net.kencochrane.raven.log4j.SentryAppender" name="Sentry">  
     <param name="dsn" value="http://[two hashes separared by a colon]@log.tangrambpm.es/5" />  
     <filter class="org.apache.log4j.varia.LevelRangeFilter">  
       <param name="levelMin" value="INFO" />  
     </filter>  
   </appender>  
   <category name="com.ecyrd.jspwiki">  
     <priority value="ERROR">  
   </priority></category>  
 ...  
   <root>  
     <priority value="INFO">  
     <appender-ref ref="CONSOLE">  
     <appender-ref ref="FILE">  
     <appender-ref ref="Sentry">  
   </appender-ref></appender-ref></appender-ref></priority></root>  
 </log4j:configuration>  

After theses steps and a Liferay restart you should be done.

References

• [1] Sentry doc main page: http://sentry.readthedocs.org/
• [2] Liferay main product page: https://www.liferay.com/
• [3] Liferay log4j configuration wiki: http://www.liferay.com/es/community/wiki/-/wiki/Main/How+to+configure+the+logs+in+Liferay/
• [4] Liferay Log4j configuration file portal-log4j.xml: https://github.com/liferay/liferay-portal/blob/6.1.x/portal-impl/src/META-INF/portal-log4j.xml
• [5] Liferay Log4j DTD file for Liferay: https://github.com/liferay/liferay-portal/blob/6.1.x/portal-impl/src/META-INF/log4j.dtd 
• [6] Nuxeo Log4j configuration example for Sentry: http://answers.nuxeo.com/questions/5914/how-to-use-third-party-log4j-appenders-in-nuxeo-tomcat-distribution
• [7] raven-java downloads: https://github.com/kencochrane/raven-java/downloads
• [8] raven-log4j  Maven repository entry at mvnrepository.com: http://mvnrepository.com/artifact/net.kencochrane.raven/raven-log4j/4.1.2

¿Por qué los polvorones y mantecados contienen E-320?

Pregunta abierta a fabricantes y distribuidores de polvorones y mantecados y especialistas de alimentación

Hasta ayer no sabía nada sobre el antioxidante E-320, pero al leer los ingredientes de los polvorones y mantecados, y en la tónica de revisar los ingredientes de los alimentos para vigilar la alimentación en nuestra familia, me he encontrado con este ingrediente que desconocía.

Según las referencias consultadas E-320 es un antioxidante sintético utilizado en la industria industria petrolífera empleado para conservar grasas y cuyo consumo debería evitarse por la posibilidad de la aparición de los siguientes efectos adversos:

• Hiperactividad
• Asma
• Urticaria
• Insomnio
• Aumento del colesterol en la sangre
• Problemas de metabolismo en el hígado
• Adormecimiento
• Tumores cancerígenos

Aunque en Europa y USA esté permitido su uso, en Japón está prohibido.

Por lo cual me gustaría saber:

• ¿Para qué hace falta añadir este ingrediente? ¿No es la manteca de cerdo y el azúcar suficiente para conservar estos alimentos?
• ¿Qué alternativas existen a E-320 para el caso de polvorones y mantecados?
• ¿Por qué no se están usando estas alternativas?

Gracias.

Atentamente,
un ciudadano a pie que le gustaría disfrutar de los dulces navideños sin preocupaciones


Referencias (todas consultadas el día 30/12/2013):

• Artículo en Wikipedia (Butil-hidroxi-anisol) (BHA): http://es.wikipedia.org/wiki/Butilhidroxitolueno
• "Aditivos alimentarios": http://aditivos-alimentarios.blogspot.com.es/2010/12/e320-butilhidroxianisol-bha.html
• "Food-info.net": http://www.food-info.net/es/e/e320.htm
• "Foot matters tv": http://foodmatters.tv/articles-1/top-10-food-additives-to-avoid
• About.com Chemistry: http://chemistry.about.com/od/foodcookingchemistry/a/bha-bht-preservatives.htm
• David Suzuki Foundation: http://davidsuzuki.org/issues/health/science/toxics/chemicals-in-your-cosmetics---bha-and-bhti/

Por internet pueden encontrarse fácilmente otras referencias.

Make Java applets work with chromium-browser on Ubuntu

The problem

On some Ubuntu versions, notably Ubuntu 12.04 LTS and some newer ones, Chromium >= 30 may not execute Java applets. In some browser versions, it will crash the browser process.

Run your own test:

http://www.java.com/en/download/installed.jsp

The result should be something like this:

The problem is a name clash with a Java library: libnet.so.

This does not happen with all Chromium versions. In recent versions, all the provided libraries can't be linked statically into one single executable as before. (The resulting binary would be too big.) Hence the libs directory.

The solution

Of course, you can use Mozilla Firefox or some other browser which allows the use of Java applets. For me, the Google Chrome browser also works correctly.

So if you still want to use Chromium, go on.

Let's change all libnet occurrences with some other unused library name.

Here we go:

sudo su
apt-get install bbe
cp -a /usr/lib/chromium-browser /usr/lib/chromium-browser.bak
cd /usr/lib/chromium-browser
for f in `rgrep -l libnet .`
do
  bbe -e 's/libnet/libxet/' $f > $f.new
  mv -f $f.new $f
done
mv libs/libnet.so libs/libxet.so
mv libs/libnet_with_v8.so libs/libxet_with_v8.so
chmod +x chromium-browser

Before running the test again make sure all chromium-browser processes have stopped and kill them if necessary:

ps aux | grep chromium-browser

killall chromium-browser




Now run the applet test again and see if it works.

Take into account that the next package update may overwrite these changes.

Conclusion

Although a bit hackish, if you have to use Java applets in daily work like me and you really want to use Chromium, here you have a solution.

Buildout errors with gocept.recipe.env

The syntom

I get a really very strange error when running buildout.

While:
  Installing nginx.
  Getting option config:PS1.
Error: The option name in substitution, ${debian_chroot:+($debian_chroot)},
has invalid characters.

One strange behavior is that it does not occur every time. I still haven't discovered exactly when this error occurs. But the most strange and surprising fact is that it corrupts the buildout control file .installed.cfg, making it unusable for later buildout runs.

The problem

After digging a while I have discovered that my buildout (incepted by co-workers) is using the gocept.recipe.env 1.0 to get a environment variable (USER), and the error is related to the PS1 environment variable. It includes a dollar sign ($) and or this recipe or buildout (2.1.0) is not escaping it properly.

The solution

I opted to separate my part into two and to use collective.recipe.environment instead.

[mypart]

recipe = gocept.recipe.env

key = value

...

is converted to:

[enviroment]

recipe = collective.recipe.environment

[mypart]

USER = ${environment:USER}

key = value

...

I hope that this solution will be less erroneous / more solid.

Enable Wifi Tethering on my MTK6589 Flying F600 with Android Jelly Bean 4.1.2

The problem

Google has introduced changes in the tethering module since Android 4.1.2 to allow carriers to charge their customers Internet usage from other devices. [1]

On the other hand, I have a less famous Android 4.1.2 mobile phone, the Flying F600, which uses the Mediatek MTK6589 Quad Core processor, so I needed a rooting kit that works with this phone and preferably with my MacBook Pro with OS X 10.8.5.

The solution

Overview

1. Download and root the phone
2. Create a script for enabling NAT using iptables
3. Download and install Script Manager
4. Configure the script to execute automatically
5. Add the tethering widget to your home screen.

Steps

1. Download and root the phone

Before I did this I had no idea, but I'm a hacking guide, so I have plenty of knowledge SSH, shell scripting, networking, etc.

The root kit which I found to work well is the one I found at droidchina forums [2]. The download link is here. Previous to downloading I had to register.

I just followed the steps of the article. The compressed zip file includes the Android Debug Bridge executable ADB for Windows, Linux y OS X.

Before running the rooting command (run.sh), make sure your phone's USB connection is debug mode. In the following screen shot you can see the spanish version.

What I found interesting is the "adb shell" command which lets you hack directly your device. In combination with "su -" you can do a lot of things. It's just a custom Linux OS:

2. Create the script for enabling NAT

After some attempts to enable NAT using iptables, I found that the following script worked best:

su -c "iptables -tnat -F"
su -c "iptables -tnat -A POSTROUTING -s 192.168.0.0/16 -o ccmni0 -j MASQUERADE"
su -c "echo 1 > /proc/sys/net/ipv4/ip_forward"

You can write the script using "adb shell" and "vi". Save the script to /sdcard. I called it "tether_enable". You could also write it using an Android Text Editor like Jota or using a Terminal with vi or some other native commands like cat, etc.

Some explanations here:

• The first line flushes iptables "nat" routing table. This is good when the script is executed several times.
• The "ccmni0" interface is my normal public 3G data services interface. You can use the "netcfg" command to find out yours. Anyway, you can try it without specifying the "-o ccmni0" option of iptables.
• I tried it with "-A natctrl_nat_POSTROUTING" found in several forums but it didn't work for me. I still don't know why. So I used the POSTROUTING chain.
• Whenever I enable or disable tethering, ip_forward is reset to 0.

3. Download and install Script Manager

Script Manager (SManager) is an excellent application to enable script execution. It allows you to execute scripts directly, mark it as favorite, execute on boot or on network change.

4. Configure the script to execute automatically

After installing it, I selected my /sdcard/tether_enable script and marked it as favorite, root and net. This executes the script whenever the network config changes.

       Select the script       

Mark as favorite,
root and net

5. Add the tethering widget to your home screen

Locate the tethering widget

Hold and drag it to
 the home screen       

Choose Wifi tether      

Widget is on home screen

Conclusions / What I learned

• Android is just a custom Linux
• Rooting your Android phone opens lots of new possibilities
• Rooting is easy
• Tethering is fixed by enabling NAT (with iptables)
• How to capture screens: use "adb 

References

[1] Android Source Code Issue Tracker, Issue 38563 

[2] Article: Auto Root MTK6589 (One click root) in Droid China forums

Parsing X509v3 certificates and PKCS7 messages with Python

Introduction

Recently I had the need to get out the following information out of certificates and PKCS7 messages:

• A certificate's validity period (notBefore, notAfter attributes)
• A PKCS7 digital signature's author and signing time

Some basics

Digital certificates are ASN.1 (Abstract Syntax Notation One) structures DER (Distinguished Encoding Rules). 

ASN.1 is something like Backus-Naur Form used for describing data structures, e.g.:

 MyType ::= SEQUENCE {  
   myObjectIdentifier OBJECT IDENTIFIER,  
   myNumbers SEQUENCE OF MyNumber,  
   myMessage VisibleString  
 }  
 MyNumber ::= INTEGER (0..255)  

Although it's nearly 30 years old (being originally part of the CCITT X.409:1984 spec), it's still often used in the Public Key Infrastructure world. For example, digital certificates like X.509 and PKCS (Public Key Cryptography Standards) make use of ASN.1. Simply said, it's a simple and common way to define data structures.

Beside native data types like booleans, integer numbers, real numbers, date-times, strings and null, ASN.1 includes keywords to build complex data types. In the above example, SEQUENCE was used to build a C like struct data structure and  SEQUENCE OF for a list of numbers between 0 and 255.
The CHOICE keyword acts much like C's union keyword. It's used to pack several alternative data structures into a same space. One very special primitive type is Object Identifier. It's used to reference an already globally registered data type o semantic interpretation. For example, the commonName value used in digital certificates Subject field of type Name has the globally registered ID: 2.5.4.3.

On the other hand, DER is a way to digitally encode ASN.1 data structures with the goal to transfer this information to some other party.

If you have a certificate in PEM format it's easy to convert them to DER with OpenSSL:

openssl x509 -in cert.pem -out cert.der -outform DER

The digital certificate data structure

The X.509v3 digital certificate data structure is quite complex. The IETF has published its format as used on the Internet, which has evolved over time: RFC 2459  ->  RFC 3280  ->  RFC 5280.

Here is the ASN.1 description of the first two hierarchy levels:

Certificate ::= SEQUENCE {  
     tbsCertificate       TBSCertificate,  
     signatureAlgorithm   AlgorithmIdentifier,  
     signatureValue       BIT STRING }  
   
TBSCertificate ::= SEQUENCE {  
     version         [0]  EXPLICIT Version DEFAULT v1,  
     serialNumber         CertificateSerialNumber,  
     signature            AlgorithmIdentifier,  
     issuer               Name,  
     validity             Validity,  
     subject              Name,  
     subjectPublicKeyInfo SubjectPublicKeyInfo,  
     issuerUniqueID  [1]  IMPLICIT UniqueIdentifier OPTIONAL,  
                          -- If present, version shall be v2 or v3  
     subjectUniqueID [2]  IMPLICIT UniqueIdentifier OPTIONAL,  
                          -- If present, version shall be v2 or v3  
     extensions      [3]  EXPLICIT Extensions OPTIONAL  
                          -- If present, version shall be v3  
     }

Well, this doesn't seem to be quite complex. Name is basically a collection of tuples (Object Identifier, Value), where:

• Object Identifier is a globally (Internet) registered identifier which you can look up in the internet, e.g. in oid-info.com. One example could be "commonName" which is 2.5.4.3.
• Value is normally a string. (There are several types of strings in ASN.1.)

The "not so trivial" part of this data structure is the extensions part which only may be present in X.509 certificates of version 3 or later. The original RFC states:

   The extensions defined for X.509 v3 certificates provide methods for
   associating additional attributes with users or public keys and for
   managing the certification hierarchy.

One of the more interesting standard extensions is the Subject Alternative Names (aka SubjectAltName) extension:

   The subject alternative names extension allows additional identities
   to be bound to the subject of the certificate.  Defined options
   include an Internet electronic mail address, a DNS name, an IP
   address, and a uniform resource identifier (URI).  Other options
   exist, including completely local definitions.  Multiple name forms,
   and multiple instances of each name form, may be included.  Whenever
   such identities are to be bound into a certificate, the subject
   alternative name (or issuer alternative name) extension MUST be used.

   Because the subject alternative name is considered to be
   definitiviely bound to the public key, all parts of the subject
   alternative name MUST be verified by the CA.

As things happen, some of our spanish officially recognized Certificate Authorities packs non-standard attributes into SubjectAltNames. The extension data is available again as a DER encoded ASN.1 data package, so that you have to feed it through the appropriate parser.

About reading X.509 digital certificates with Python

Now, we already know that X.509 certificates are ASN.1, DER-encoded data structures. Thanks to the excellent PyASN1 library we can read those data structures. But something is still missing. DER encoded ASN.1 data packages are not self describing, i.e. we must have a data structure description, just like a C typedef struct or a Python class definition.

It would be great to have a ASN.1 to PyASN1 compiler. Then we could pick up the X509v3 ASN.1 description and translate it to Python. Until recently there was none but now there is an attempt to fill this gap: asn1ate. Before, most existing data model descriptions for PyASN1 were translated by hand. The separate PyASN1-modules package includes common data structures like PKCS12, X509v3 (RFC2459), etc.

Here is one example: the PKCS12 data structure translated to PyASN1:

#
# PKCS#12 syntax
#
# ASN.1 source from:
# ftp://ftp.rsasecurity.com/pub/pkcs/pkcs-12/pkcs-12.asn
#
# Sample captures could be obtained with "openssl pkcs12" command
#
from pyasn1.type import tag, namedtype, namedval, univ, constraint
from pyasn1_modules.rfc2459 import *
from pyasn1_modules import rfc2251
class Attributes(univ.SetOf):
    componentType = rfc2251.Attribute()
class Version(univ.Integer): pass
class CertificationRequestInfo(univ.Sequence):
    componentType = namedtype.NamedTypes(
        namedtype.NamedType('version', Version()),
        namedtype.NamedType('subject', Name()),
        namedtype.NamedType('subjectPublicKeyInfo', SubjectPublicKeyInfo()),
        namedtype.NamedType('attributes',
            Attributes().subtype(implicitTag=tag.Tag(
                tag.tagClassContext, tag.tagFormatConstructed, 0)))
    )

Hands on with PyASN.1

Now, let's try parse a certificate. You can find the test certificate used in this example in the pyx509 package described below. You can also generate your own certificate with OpenSSL:

openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.der -days 1000 -outform DER -nodes

First, we'll install pyasn1 and pyasn1-modules:

$ sudo pip install pyasn1
Downloading/unpacking pyasn1
  Downloading pyasn1-0.1.7.tar.gz (68kB): 68kB downloaded
  Running setup.py egg_info for package pyasn1
    
Installing collected packages: pyasn1
  Running setup.py install for pyasn1
    
Successfully installed pyasn1
Cleaning up...
$ sudo pip install pyasn1-modules
Downloading/unpacking pyasn1-modules
  Downloading pyasn1-modules-0.0.5.tar.gz
  Running setup.py egg_info for package pyasn1-modules
    
Requirement already satisfied (use --upgrade to upgrade): pyasn1>=0.1.4 in /Library/Python/2.7/site-packages (from pyasn1-modules)
Installing collected packages: pyasn1-modules
  Running setup.py install for pyasn1-modules
    
Successfully installed pyasn1-modules
Cleaning up...

Now we'll go ahead and read a certificate:

$ python
>>> from pyasn1.codec.der.decoder import decode
>>> from pyasn1_modules import rfc2459
>>> derData = file('cert.der', 'rb').read()
>>> cert, rest = decode(derData, asn1Spec=rfc2459.Certificate())
>>> print cert.prettyPrint()
Certificate:
 tbsCertificate=TBSCertificate:
  version='v3'
  serialNumber=1019333950
  signature=AlgorithmIdentifier:
   algorithm=1.2.840.113549.1.1.5
   parameters=0x0500

  issuer=Name:
   =RDNSequence:
    RelativeDistinguishedName:
     AttributeTypeAndValue:
      type=2.5.4.6
      value=0x13024553
    RelativeDistinguishedName:
     AttributeTypeAndValue:
      type=2.5.4.10
      value=0x1304464e4d54
    RelativeDistinguishedName:
     AttributeTypeAndValue:
      type=2.5.4.11
      value=0x130f464e4d5420436c6173652032204341

  validity=Validity:
   notBefore=Time:
    utcTime=100903074356Z

   notAfter=Time:
    utcTime=130903074356Z

  subject=Name:
   =RDNSequence:
    RelativeDistinguishedName:
     AttributeTypeAndValue:
      type=2.5.4.6
      value=0x13024553
    RelativeDistinguishedName:
     AttributeTypeAndValue:
      type=2.5.4.10
      value=0x1304464e4d54
    RelativeDistinguishedName:
...

Not bad for the first attempt. But nearly all attributes seem to be encoded. Let's get the subject and see if we can transform it to a readable string.

>>> cert = cert['tbsCertificate'] # just get the core part of the certificate
>>> subject = cert['subject']
>>> rdnsequence = subject[0] # the subject is only composed by one component
>>> for rdn in rdnsequence:
...    oid, value = rdn[0]  # rdn only has 1 component: (object id, value) tuple
...    print oid, ':', str(value)

...

2.5.4.6 : ES
2.5.4.10 : FNMT
2.5.4.11 : FNMT Clase 2 CA
2.5.4.11 :  703002474
2.5.4.3 : 8NOMBRE REVILLA DERKSEN ALEJANDRO ERNESTO - NIF ...

Now we have some readable output. The Object Identifiers have the following meaning:

• 2.5.4.6: countryName, abbreviated: C
• 2.5.4.10: organizationName, abbreviated: O
• 2.5.4.11: organizationalUnitName, abbreviated: OU
• 2.5.4.3: commonName, abbreviated: CN

With OpenSSL, it is normally displayed like this:

$ openssl x509 -in ub1204/svn/pyx509/exampledata/cert.der -inform DER -subject -noout
subject= /C=ES/O=FNMT/OU=FNMT Clase 2 CA/OU=703002474/CN=NOMBRE REVILLA DERKSEN ALEJANDRO ERNESTO - NIF ...




Using pyx509 to parse X.509 certificates

The pyx509 library is an attempt to offer a more Python like data structure. It brings it's own model of X.509 for PyASN1. My fork of pyx509 includes the possibility to parse / display SubjectAltName directory name (dirName) name parts. Sorry, no still no PyPi / setup.py, so you have to download the zip/tar ball and uncompress it.

Here an example:

./x509_parse exampledata/cert.der
=== X509 Certificate ===
X.509 version: 3 (0x2)
Serial no: 0x3cc1cd3e
Signature algorithm: SHA1/RSA
Issuer: C=ES, O=FNMT, OU=FNMT Clase 2 CA
Validity:
 Not Before: 2010-09-03 07:43:56
 Not After: 2013-09-03 07:43:56
Subject: C=ES, CN=NOMBRE REVILLA DERKSEN ALEJANDRO ERNESTO - NIF ..., O=FNMT, OU=703002474, OU=FNMT Clase 2 CA
Subject Public Key Info:
 Public Key Algorithm: RSA
  Modulus: (b64)
...

  Exponent: 65537

Extensions:
...

 Subject Alternative Name: is_critical: False
  email: ernesto.revilla@gmail.com
  dirName: Apellido1=REVILLA, Apellido2=DERKSEN, DNI=..., Nombre=ALEJANDRO ERNESTO
...

=== EOF X509 Certificate ===

This seems to give us a much more usable output and may be a good alternative to parsing OpenSSL output.

Displaying digital signatures / timestamps with pyx509

With pyx509 we can also display some data of digital signatures complying PKCS7:

= PKCS7 signature block =
PKCS7 Version: 1
== Encapsulated content Info ==
ContentType: data
Content: None
== Signer info ==
Certificate serial number: 0x89bbba0749918db3
Issuer: C=es, ...
Digest Algorithm: SHA-1
Signature: (b64)
 gXpU5jadSY+FVBoeCdvn1/m5bzEMzN3ZKuiN9sPk79iJgX+DDDOMH6K5Scnh
 wLL7nHRT983GlhTY1A2QE1VryWTbuBGK08oalKIM8QZs3UfZa5dXsx83eS4b
 /M/icfIf6CHu1fWZ4VBJ4mva2N3nh2r0FV09bvuj1bodl4kXJAs=
Attributes:
     contentType: data
     serialNumber: 0x89bbba0749918db3
     signingTime: 2011-10-04 14:36:51
     messageDigest: y8OX3qoZBY4/Cc6/w0xuRqzzQzU=
     signingCertificate: 0x89bbba0749918db3
== EOF Signer info ==
=== X509 Certificate ===
X.509 version: 3 (0x2)
Serial no: 0x89bbba0749918db3
Signature algorithm: SHA1/RSA
Issuer: C=es, ...
Validity:
 Not Before: 2011-09-17 00:00:00
 Not After: 2031-09-12 11:09:54
Subject: C=es, CN=REVILLA DERKSEN, ALEJANDRO ERNESTO...
Subject Public Key Info:
 Public Key Algorithm: RSA
  Modulus: (b64)
   AOs2/Pip46F5BJPBQd/5bwS1HO97lJ74ZjJfGtvEH831d6Ld4bsF9jdFOjlx
   mv+kxYNFryZZFWM109+zng/PiU8NZPRZt4XlTO7qb3r2g5AR17EQWJNokQto
   s3w3cXSEDPxxFmTHEhGarTLddEg2o1v9/UIlMS8mzHej0Q9uBuuh
  Exponent: 65537

Extensions:
 Authority Key Id Ext: is_critical: False
  key id: (b64)
   AiuDvGb4bxWnCsZJ9/RHNrRhSxk=
 Basic Constraints Ext: is_critical: False
  CA: False
  max_path_len: None
 Subject Alternative Name: is_critical: False
  email: tramitacion.electronica@telefonica.es
  dirName: Apellido1=REVILLA, Apellido2=DERKSEN, DNI=..., Nombre=ALEJANDRO ERNESTO
 Subject Key Id: is_critical: False
  key id: (b64)
   Zh0L6JJSz+GgiCimE4U7s5PHH+g=
Signature: (b64)
 k1OVoQyNZv0ASor/bitI6JgJm37piIheIzwdKSgEtKeQuIXfA5V5rclPVUg7
 PW71JTQyY8iDbvJB4sb4FH5XyjOXUmf3CXiG7ppS48cQXSf1k3wHWZB0neTE
 V3XxZnPjqWvv0x0ScsOGKxpHjyy8SFZMKR6tnfQ4TXfHMxid7dw=
=== EOF X509 Certificate ===

We can clearly see that there is one signature block (Signer Info) which specifies the original message's digest, the digest algorithm used (SHA-1), the signature, a reference to the certificate, the certificate itself and the signing time.

Here one example for a time stamp token gotten from a public Time Stamp Authority (TSA):
./pkcs7_parse.py exampledata/timestamp.tst = PKCS7 signature block = PKCS7 Version: 3 == Encapsulated content Info == ContentType: TimeStampToken === Timestamp Info === Version: 1 Policy: 1.3.4.6.1.3.4.6 msgImprint: Algorithm Id: 1.3.14.3.2.26 Value: (b64) rnLdD3molzRsebPvq7oOSG9n8fU= Serial number: 134059559 Time: 20131011084712Z ==== Accuracy ==== Seconds: 1 Milis: 1 Micros 2 ==== EOF Accuracy ==== TSA: === EOF Timestamp Info === == Signer info == Certificate serial number: 0x5079e Issuer: C=ES, CN=MINISDEF-EC-WPG, O=MDEF, OU=PKI Digest Algorithm: SHA-1 Signature: (b64) ...

Attributes: contentType: TimeStampToken messageDigest: KpRSk0vbBke+8G40MIII9NNb51E= signingCertificate: 0x5079e == EOF Signer info == === X509 Certificate === X.509 version: 3 (0x2) Serial no: 0x5079e Signature algorithm: SHA1/RSA Issuer: C=ES, CN=MINISDEF-EC-WPG, O=MDEF, OU=PKI Validity: Not Before: 2011-08-17 09:50:22 Not After: 2021-08-17 09:50:22 Subject: C=ES, CN=Sello de tiempo TS@ - @firma - desarrollo, O=MDEF, OU=PKI, serialNumber=S2833002E Subject Public Key Info: Public Key Algorithm: RSA Modulus: (b64) ...

Exponent: 65537 Extensions: ...

Extended Key Usage: is_critical: True timeStamping Key Usage: is_critical: True digitalSignature,nonRepudiation Subject Alternative Name: is_critical: False email: soporte.afirma5@mpt.es dirName: CN=TS@- Autoridad Sellado de tiempo-desarrollo, O=Ministerio de la Política Territorial y Administración Pública, certType=sello de tiempo, serialNumber=S2833002E ...

=== EOF X509 Certificate === = EOF PKsCS7 signature block = 

Conclusions

Although pyx509 is rather incomplete it may fulfill your needs and may be an alternative to parsing certificates, digital signatures and timestamps with OpenSSL.

Changing Alfresco object properties with python + cmislib

The problem

Some code saved some wrong (custom) property values but you have not the XML extensions to edit the values with Alfresco-Explorer or Alfresco Share.

One possible solution: Use Python and cmislib

If you haven't easy_install or python pip, please install this first. For Ubuntu/Debian this would be:

sudo apt-get install python-pip

So here we go:

sudo pip install cmislib
python

import cmislib
URL = 'http://localhost:8080/alfresco/cmis'
USERNAME = 'admin'
PASSWORD ='admin'
client = cmislib.CmisClient(URL, USERNAME, PASSWORD)
repo = client.getDefaultRepository()
doc = repo.getObjectByPath('/path/to/doc_or_folder')
doc.updateProperties({'propertyname': 'propertyvalue'})

The property value should have changed.

Memory and CPU hotplug on Ubuntu Guests and VMWare ESXi hosts (including vCloud)

Introduction

With some cloud projects in mind and some virtual hardware estimations I was curious to see if it's possible to hot plug virtual memory and CPU to Ubuntu guests. Here is what I discovered.

• Infrastructure: vCloud Director on Acens.com (VMWare ESXi)
• Guest: Ubuntu server 10.04 (kernel 2.6.32-45-server #100-Ubuntu SMP Wed Nov 14 11:02:27 UTC 2012 x86_64 GNU/Linux)
• virtual HW version: 7
• vCPU: 1
• vMemory: 1 GB

Hot add virtual CPUs

Check current CPUs:

# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model  : 37
model name : Intel(R) Xeon(R) CPU           L5640  @ 2.27GHz
stepping : 1
cpu MHz  : 2266.747
cache size : 12288 KB
fpu  : yes
fpu_exception : yes
cpuid level : 11
wp  : yes
flags  : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc up arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm arat
bogomips : 4533.49
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

Now, add the virtual CPUs directly from the vCloud control panel (from the hardware tab of the VM properties). Initially, these new CPUs are offline. To active each new CPU we have to:

echo 1 > /sys/devices/system/cpu/cpuX/online

where X is the number of the new CPU.

Check that new CPUs are online:

# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model  : 37
model name : Intel(R) Xeon(R) CPU           L5640  @ 2.27GHz
stepping : 1
cpu MHz  : 2266.747
cache size : 12288 KB
fpu  : yes
fpu_exception : yes
cpuid level : 11
wp  : yes
flags  : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc up arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm arat
bogomips : 4533.49
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model  : 37
model name : Intel(R) Xeon(R) CPU           L5640  @ 2.27GHz
stepping : 1
cpu MHz  : 2266.747
cache size : 12288 KB
fpu  : yes
fpu_exception : yes
cpuid level : 11
wp  : yes
flags  : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc up arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm arat
bogomips : 4533.49
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

You can use the script provided in this article:

http://communities.vmware.com/docs/DOC-10493

This enables automatically all new vCPUs.

Hot add virtual memory

This operations is analogue to one described above, but the kernel module that detects the new memory is not automatically loaded (at least in my Ubuntu install), so we have to do it manually:

# modprobe acpi_memhotplug

Let's check current memory first:

# free -m
             total       used       free     shared    buffers     cached
Mem:           997        112        885          0         10         42
-/+ buffers/cache:         59        937
Swap:         2015          0       2015

Add the memory through vCloud (or VMWare vSphere Infrastructure Cliente - VCLI). Although the kernel will have detected the new memory, it's still not available. We have to enable it in the same way we did it before with virtual CPUs:

echo 1 > /sys/devices/system/memory/memoryX/online

where X is the number of the new memory block.

After the memory blocks have been enabled, we can check the available memory:

# free -m
             total       used       free     shared    buffers     cached
Mem:          1893        141       1752          0         10         42
-/+ buffers/cache:         88       1805
Swap:         2015          0       2015

As with CPU hot plug, you can use the script provided in this article:

http://communities.vmware.com/docs/DOC-10492

This enables automatically all new memory blocks.

Epilogue

It's easy to add new resources on the fly but it may not be that easy to remove them.

Compile Nginx with custom OpenSSL in Ubuntu 10.04

The problem

We need Server Name Indication (SNI) for nginx, but the OpenSSL version included in Ubuntu 10.04 does not support it.

Edit: I just noticed that I was wrong. SNI  *IS* supported by the version of openssl provided with Ubuntu 10.04:  libssl0.9.8-7ubuntu8.13 

The solution

• Download, build and install a recent version of OpenSSL
• Compile nginx against this version of OpenSSL
• Do not replace the system openssl

Here are the steps I took:

# install dependencies
sudo apt-get install build-essential libpcre3-dev libxml2-dev libxslt1-dev
cd
mkdir -p src; cd src
wget http://www.openssl.org/source/openssl-1.0.1c.tar.gz
tar xzf openssl*
cd openssl*
./config shared zlib-dynamic
make
# This installs everything in /usr/local/ssl

sudo make install

wget http://nginx.org/download/nginx-1.2.5.tar.gz
tar xzf nginx*
cd nginx*
# edit auto/lib/openssl/conf manually or use sed
sed -i -e 's|\.openssl/||' auto/lib/openssl/conf
./configure --with-openssl=/usr/local/ssl --with-http_ssl_module
make
# test if SNI is displayed
./objs/nginx -V
# if everithing is ok, install
sudo make install

References

• Nginx: Configuring HTTPS servers

Restoring iPhone under Ubuntu 12.04 / VirtualBox / WinXP / iTunes

Today, I bricked my iPhone and here goes my experience.

My plataform: Ubuntu 12.04/amd64, Oracle VirtualBox 4.2.0 / WinXP SP 2, iTunes 10.x

Some block posts reported permission problems of USB device files and adviced starting VirtualBox as root. And so I did. But the restoration / update process stuck.

Problem 1: VirtualBox does not recognize when a new USB device is plugged in. This may be due to starting VirtualBox as root, but I'm not sure, so stop or hibernate guest OS, do /etc/init.d/vboxdrv restart, and run guest again.

Problem 2: VirtualBox stops anywhere in the update process. There seems to be a problem with constant USB device switching (iPhone recovery mode), etc. It was just a matter of disconnecting the USB device with help of VirtualBox USB menu. It gets automatically reconnected and the update process continues.


I'll see if I find a cleaner way to do this.

No uses variables globales

Este post explica por qué mejor no usar variables globales en entornos Web y está dirigido a personas que aún no lo ven tan claro. Para este ejemplo emplearé Apache, mod_python/mod_wsgi, Django y Memcached pero cualquier otro entorno Web también sirve.

Apache

Apache tiene 2 modos de funcionamiento:
1. Prefork: Mediante procesos
2. Worker: Mediante hebras y procesos
Más frecuentemente se suele usar el modo 1 porque es más seguro ya que no todas las librerías usadas están escritos teniendo en cuenta que pueden ser empleadas por varias hebras a la vez. Se dice que no son thread-safe.

En las variantes de Unix (Linux, FreeBSD, NetBSD, Mac OS X, Unix, Solaris, etc.), el modo 1 se basa en una llamada al sistema operativo fork que divide un proceso en dos, uno considerado "padre" y otro "hijo". Ambos procesos tienen una copia idéntica de todos los datos. Pero una vez divididos, los cambios en memoria realizados por un proceso ya no afectan al otro.

En caso de Apache, existe un proceso maestro responsable de crear los hijos, controlar el número de procesos disponibles, etc. Los procesos hijos son los que atienden a las peticiones. En caso de los intérpretes de lenguajes o en general cualquier módulo, se carga después de realizar el fork.

mod_python y mod_wsgi

mod_python y mod_wsgi permiten el uso de más de un intérprete si Apache está sirviendo varias aplicaciones Web y pueda haber interferencias entre ellas. Si, por ejemplo, usamos dos intérpretes y tenemos diez procesos de Apache, podemos tener hasta 2 x 10 = 20 intérpretes de Python activos a la vez. Al mostrar los procesos con ps o top no aparecen como Python sino como Apache porque se ha invocado el intérprete desde una llamada a una librería. Por tanto, se ejecuta en el espacio de memoria y con los permisos del proceso Apache.

Por otra parte, todo lo que un proceso almacena en variables globales, por ejemplo datos de una base de datos, datos pre-calculados o páginas web completas, no puede ser aprovechado por otro proceso. Supongamos que haya que cargar un conjunto de datos de tamaño considerable y lo almacenamos en variables globales. En caso de Python, por ejemplo, podría ser un diccionario asignado a una variable con ámbito de módulo durante el inicio de la aplicación o cuando el usuario realice una determinada petición. Cada proceso tendría entonces que seguir los mismos pasos duplicando así los datos en memoria y realizando las mismas consultas a la base de datos. Esto no es demasiado óptimo y hará que cada proceso de Apache ocupe mucha memoria.

¿Qué podemos hacer al respecto?

Memcached

La solución es usar una caché compartida entre los procesos e incluso entre varias máquinas. Siempre que queramos aprovechar un dato elaborado, tal como los resultados de una consulta de base de datos, una página completa, un cálculo estadístico, etc. lo almacenamos en la caché compartida para que cualquiera de los procesos Apache pueda aprovecharlo.

Memcached es una caché de este tipo. Básicamente, se trata de un software que permite almacenar y recuperar conjuntos de datos desde cualquier ubicación de nuestra red.

La lógica podría ser siempre la misma:
1. Intentar obtener el valor requerido desde la caché
2. En caso de no existir, lo calculamos y lo almacenamos en caché.
3. Hacer lo oportuno con el valor.

Un fragmento Python podría ser:

value = cache.get(key)
if value == None:
    # calc value
    value = do_calc_value_here()
    cache.set(key, value)
# do something with value
render_template(template, value)

Todos los sistemas de caché emplean una clave para almacenar y poder recuperar con posterioridad el valor. A la hora de establecer la clave es importante indicar si el valor cacheado puede compartirse entre todos los usuarios o no. Por otra parte, es importante que se actualice la caché cuando algún usuario provoque un cambio, es decir, que la caché no ofrezca valores inconsistentes.

Django & Memcached

Tomemos como ejemplo Django. Podemos almacenar un valor recuperado de la base de datos siempre que la petición de otra persona no lo actualice ni que exista otra aplicación que actualice los mismos datos directamente sobre la base de datos. Para no tener que lidiar con diferentes APIs de los sistemas de caché, Django ofrece una abstracción:

from django.core.cache import cache

cache.set(clave, valor, tiempo) # almacenar un valor en cache
valor = cache.get(clave)        # recuperar un valor de cache

Para facilitar el uso con página completas, puede emplearse el decorador cache_page:

from django.views.decorators.cache import cache_page

@cache_page(60 * 15)
def my_view(request):
    ...

Esto almacena la página generada durante 900 segundos y emplea la versión cacheada si está disponible. La función cache_page hace toda la mágica.

Supongamos, sin embargo que la página dependa de la persona que realiza la consulta:

from django.core.cache import cache

def my_view(request):
    key = "my_view" + request.user.username
    page = cache.get(key)
    if not page:
        page = render....
        cache.set(key, page, 60 * 15)
    return page

El valor a almacenar en caché ha de ser "persistible". El cliente de memcached para python (OJO: existen varios clientes en la actualidad: python-memcachedmemcached, python-libmemcached y pylibmc usa pickle, siempre que no se trate de una cadena de caracteres. Esto significa que no podemos cachear objetos tipo conexión a base de datos, objeto sesión, etc.

Es importante dimensionar el tamaño de RAM asignado a memcached, así como el número de posibles conexiones. Los valores por defecto son 64 MBytes (parámetro -m) y 1024 (parámetro -m) respectivamente. Mire la ayuda de memcached para ver cómo modificar estos parámetros (memcached -h).

Django ofrece incluso la posibilidad de cachear trozos de una plantilla:

{% load cache %}
{% cache 500 topmenu %}
    .. topmenu ..
{% endcache %}

En caso que el menú dependa del usuario podría usarse el siguiente fragmento:

{% load cache %}
{% cache 500 topmenu request.user.username %}
    .. topmenu ..
{% endcache %}

Es decir, todos los parámetros a partir del segundo de la etiqueta de plantilla "cache" son usados para formar la clave de caché.

Finalmente, pueden almacenarse también las sesiones en memcached. Para ello indicamos lo siguiente en settings.py:

SESSION_ENGINE = "django.contrib.sessions.backends.cache"

Si deseamos que las sesiones sobrevivan el reinicio de memcached a costa de un muy pequeña reducción de rendimiento debemos usar siguiente línea:

SESSION_ENGINE = "django.contrib.sessions.backends.cached_db"

Los detalles sobre el uso de caché en Django los podemos encontrar en la Web de Django.

Conclusión

En vez de usar variables globales podemos emplear una caché compartida para ahorrar recursos y aumentar la escalabilidad. El uso de la API de caché de Django es extremadamente simple.

Alfresco Export Tool

I recently found the problem that Alfresco doesn't provide a shell script to export repository contents anymore. So I rewrote one of them found on Alfresco forums. One of the problems that I encountered is that it tries to start the VTI server module (if installed), colliding with the running Alfresco instance. Setting port to 0 disabled it.

See my post to Alfresco forums.

oodiff reloaded

Although there exists alread oodiff, it's a bit limited. What's about any format or image changes not reflected in the diff?

I had a oodiff before, but some day it stopped working. Now I renamed it to ooodiff.
Thanks to some notes from here I corrected it. This version should work well with subversion. Here's the code (PEP8/Pyflakes compatible):

#!/usr/bin/python
# Requires python-uno
# Code originally gotten from:
#   http://people.warp.es/~xtor/blog/?p=166
# See also: (google for "openoffice uno compare documents")
#   http://win32com.goermezer.de/content/view/193/274/
#   http://nxsy.org/blog/447.html
#   http://www-verimag.imag.fr/~moy/opendocument/


PORT = 2526
OOFFICE = 'ooffice'

# General imports
import filecmp
import os
import shutil
import sys
import time

# pyuno imports
import uno
from com.sun.star.connection import NoConnectException


class OOUnoConnectionError(Exception):
    pass


def getRevision(filename, revision, save_as):
    """Export a revision of a document."""
    os.system('svn export -r%s "%s" "%s"' % (revision, filename, save_as))
    return save_as


def compare(filename, oldrev='', newrev='', force=False):
    """Compare two revisions of a document."""
    if not oldrev:
        oldrev = 'BASE'
    newfilename = '%s%s%s%s' % (filename, '.' + oldrev,
        '.' + (newrev or 'workingcopy'), '.diff.tmp')
    if not newrev:
        shutil.copyfile(filename, newfilename)
    else:
        getRevision(filename, newrev, newfilename)
    oldfilename = '%s%s%s' % (filename, '.' + oldrev, '.tmp')
    getRevision(filename, oldrev, oldfilename)
    if not force and filecmp.cmp(newfilename, oldfilename):
        print ("The files seem to be the same. "
            "There are no changes. Use --force run the diff in OO.")
    else:
        oo = OO()
        doc = oo.open(newfilename)
        oo.compareCurrentDocument(doc, oldfilename)


class OO:
    """A small class to abstract OpenOffice application."""
    def __init__(self, filename=''):
        # start OO
        print "Starting OO"
        os.system("soffice -nodefault "
            "'-accept=socket,host=localhost,port=%s;urp;'" % PORT)

        # Get the uno component context from the PyUNO runtime
        localctx = uno.getComponentContext()
        # Create the UnoUrlResolver on the Python side.
        resolver = localctx.ServiceManager.createInstanceWithContext(
               "com.sun.star.bridge.UnoUrlResolver", localctx)

        cnxstr = ("uno:socket,host=localhost,port=%s;urp;"
            "StarOffice.ComponentContext")
        cnxstr = cnxstr % PORT
        # try to connect to OO
        ctx = None
        retries = 5
        while retries:
            retries -= 1
            try:
                ctx = resolver.resolve(cnxstr)
            except NoConnectException:
                time.sleep(1)
        if not ctx:
            raise OOUnoConnectionError("Can't connect to OpenOffice")

        # Get the ServiceManager object
        smgr = ctx.ServiceManager

        # Create the Desktop instance
        desktop = smgr.createInstance("com.sun.star.frame.Desktop")
        # save objects
        self.desktop = desktop
        self.smgr = smgr
        self.ctx = ctx
        if filename:
            self.open(filename)

    def open(self, filename, **kwargs):
        """Open a file."""
        print "Opening: %s" % filename
        properties = []
        for key, value in kwargs.items():
            properties.append(self.getProperty(key, value))
        properties = tuple(properties)
        doc = self.desktop.loadComponentFromURL(
            self.convertToURL(filename), "_blank", 0, properties)
        return doc

    def compareCurrentDocument(self, doc, filename):
        """Compare the current document to the specified one."""
        print "Comparing to: %s" % filename

        # Sometimes, after opening OO or loading a doc
        # we've still not access to the dispatcher or current frame
        # so we wait here a bit.
        # Get the dispatcher
        dispatcher = self.smgr.createInstance("com.sun.star.frame."
            "DispatchHelper")

        # Show tracked changes and compare documents
        frame = doc.getCurrentController().getFrame()
        property = self.getProperty('URL', self.convertToURL(filename))
        dispatcher.executeDispatch(frame, ".uno:CompareDocuments", "", 0,
            (property,))
        property = self.getProperty("ShowTrackedChanges", True)
        dispatcher.executeDispatch(frame, ".uno:ShowTrackedChanges", "", 0,
            (property,))

    def getProperty(name, value):
        """Read a OO property."""
        prop = uno.createUnoStruct("com.sun.star.beans.PropertyValue")
        prop.Name, prop.Value = name, value
        return prop
    getProperty = staticmethod(getProperty)

    def convertToURL(filename):
        """Convert a local filename to URL required by OO."""
        return uno.systemPathToFileUrl(os.path.abspath(filename))
    convertToURL = staticmethod(convertToURL)


if __name__ == '__main__':
    oldrev = newrev = ''
    force = '--force' in sys.argv
    filename = ''
    for arg in sys.argv[1:]:
        if arg.startswith('-r'):
            if not ':' in arg:
                arg += ':'
            oldrev, newrev = arg[2:].split(':')
        elif not arg.startswith('-'):
            filename = arg
    if filename:
        compare(filename, oldrev, newrev, force)
    else:
        print """
Usage: oodiff [--force] [-r[:]] 
    : the file to comparte
    --force: force comparison, although files seem the same
    -r : same as subversion. If not specified, -rPREV assumed.

    This command:
    - may not work if OpenOffice.org is already running (because
      it has to start with a listening port for UNO to work).

    - will create some temporary files (*.tmp). You have delete them
      manually when you are finished.
"""

Tomcat 6: Session replication for failover

Summary: there is a bug in tomcat 6.0.20 which inhibits tomcat to send multicasts between instances, failing to form the cluster and hence not replicate sessions.

For the current project, I have to cluster Alfresco 3.2r Enterprise. We have a mini 2-node cluster. Although hibernate L2 cache replication works correctly (you have to rename ehcache-custom.xml.sample.cluster, which was not totally clear after reading the documentation), I went for session replication, which the docs state as supported. Now, only the session replication was missing. (There seems to be a bug which makes session replication fail, but I had no time to verify it. Anyway, I wanted to go ahead and learn how to configure tomcat for session replication and fail-over.)

As I had not much idea of configuring tomcat, I picked up an existing tomcat 6.0.20 instance and a small session example. The I configured tomcat, following the session replication / cluster how-to. Finally, I copied the tomcat instance and changed any colliding ports.

But I was not able to make it work. I was looking into the log for any message about my instances following the cluster, but without luck. After trying other ports, reconfiguring the network to support multicast ping (icmp), googleing around, reading a lot of docs, etc. I found a email message (which I can't find anymore), suggesting that there is a bug in tomcat-6.0.20 not sending multicasts for cluster instance detection!

I downloaded immediately a new version (6.0.29) and configured the two instances. It worked at the first attempt.

I use Apache proxy_balancer to test the instances. Here goes my Apache config file:

<Location /balancer-manager>
  SetHandler balancer-manager
</Location>

<Proxy balancer://ajpCluster>
  BalancerMember ajp://localhost:8809 route=jvm1
  BalancerMember ajp://localhost:8810 route=jvm2
</Proxy>

<Location /sessiontest>
  ProxyPass balancer://ajpCluster/sessiontest stickysession=JSESSIONID nofailover=off
</Location>

<Location /favicon.ico>
  ProxyPass balancer://ajpCluster/favicon.ico
</Location>

The "route" parameter of BalanceMember adds just its value to the session id. The /balancer-manager url helps you to debug the cluster, displaying if both instances accept requests, how may have been processed, and to enable or disable any instances. As we can see here, my tomcat instances are listening for AJP requests on ports 8809 and 8810.

Here goes the interesting part of my conf/server.xml of both (the have just different ports):

<Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">

      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
         channelSendOptions="8">
         <Manager className="org.apache.catalina.ha.session.DeltaManager"
                  expireSessionsOnShutdown="false"
                  notifyListenersOnReplication="true"/>

         <Channel className="org.apache.catalina.tribes.group.GroupChannel">

            <Membership className="org.apache.catalina.tribes.membership.McastService"
                        address="228.0.0.4"
                        ttl="15"
                        port="45564"
                        frequency="500"
                        dropTime="3000" />

            <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                      address="auto"
                      port="4200"
                      autoBind="100"
                      selectorTimeout="5000"
                      maxThreads="6" />

            <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
                <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/> 
            </Sender>

            <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>

            <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>

         </Channel>
         <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
                filter=".*\.gif;.*\.js;.*\.jpg;.*\.htm;.*\.html;.*\.txt;" />

         <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
                   tempDir="/tmp/war-temp/"
                   deployDir="/tmp/war-deploy/"
                   watchDir="/tmp/war-listen/"
                   watchEnabled="false" />

         <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>

         <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
      </Cluster>
      ...
</Engine>

Take into account that you just have to change all enabled connector (HTTP, HTTPS, AJP, SHUTDOWN) ports and the cluster message Receiver port (4200, in the example above) for the tomcat instances which work in the machine.