10 ene. 2016

Escaping single quotes in bash

The problem

We want to define an alias for deleting python .pyc files. The alias definition:
alias rmpyc="find . -name '*.pyc' -delete
does not work correctly

Solution

Combining the bash rules for quoting:
  • Any variable and pattern is escaped whenever it is NOT enclosed within single quotes
  • Backslash escaped single quotes NOT enclosed within single quotes produce literal single quotes
  • Expressions with different quotes can be directly combined, e.g.:
echo "Double quotes: "'"'' and single quotes: '"'" 
Applying both rules we can write:
1. alias rmpyc='find . -name '"'"'*-pyc'"'"' -delete'
or
2. alias rmpyc="find . -name '"'*.pyc'"' -delete"
or
3. alias rmpyc='find . -name '\''*-pyc'\'' -delete'

Detailed explanations

  1. We start using single quotes because we want to define a literal. To obtain a literal single quote, we stop quoting the first literal (find . -name) and concatenate a single quote enclosed within double quotes, then we write the file pattern within single quotes to avoid its expansion. Again, we concatenate a single quote enclosed within double quotes and finally we add the rest of the string ( -delete).
  2. We start with double quotes while there is no pattern to be expanded. This allows us to include literal single quote. After we ended the first string (after the first literal single quote), we concatenate the file pattern enclosed within single quotes to avoid file pattern expansion. Finally we add the rest of the string within double quotes (starting with the second literal single quote).
  3. We start with single quotes like in 1. until we want a literal single quote. To get it, we end the first string (with a single quote) and then we write an escaped single quote (backslash single quote). We continue with the file pattern enclosed within single quotes to avoid its expansion. Then we add the second escaped single quote like before. Finally we concatenate the rest of the string.

11 nov. 2015

Hints for Java JMX monitoring (for Tomcat, Alfresco, Liferay, etc.)

The problem

We want to monitor an Alfresco server which is not directly accesible from the outer world. It sits inside a (VMWare) private virtual network behind a firewall.

Have you ever tried to access JMX in private virtual nets behind a firewall? 

It's not easy at all, because of the way JMX connection establishment works: The client connects to a well know RMI registry host:port. If no additional variables are set, the Java VM does these things:
1. Guess it's own IP, based on the hostname and /etc/hosts.
2. Allocate dynamically a port to receive "server" connections
3. Send the data to the client, so it can do the connection.

In our scenario (not directly reachable Alfresco server in a private virtual network), this is a real nightmare.

Here are some solutions.

Solution 1: Create / use VPN

I don't have this solution at hand, so I'll jump to the next one.

Solution 2: Fix and expose JMX ports to the outer world (protected by firewall)

1. Download Apache-Tomcat's extra catalina-jmx-remote.jar for your version of Tomcat and drop it into the tomcat/lib folder

2. Add to tomcat/conf/server.xml something like this:

<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" rmiRegistryPortPlatform="8555" rmiServerPortPlatform="8556"/>

3. Add the following variables to tomcat/bin/setenv.sh (or tomcat/scripts/ctl.sh, in case of Alfresco):
CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote "
CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
CATALINA_OPTS="$CATALINA_OPTS -Djava.rmi.server.hostname=`hostname`"

4. Open your firewall for the given ports and source IPs.
Notes:
  • The java.rmi.server.hostname value is sent verbatim to the client. This makes it possible that the hostname resolves at the server to one IP and at the client to another.
  • We disable SSL, we're supposing that the access is protected  through firewall.
  • We suppose that the firewall maps the ports for the outer world to the server in out private network
  • In most articles we can find in Internet,  java.rmi.server.hostname should be a valid IP, but this is just another reason why it's so difficult to get the right configuration. I inspected network packets with ngrep and found that the value is send verbatim.
  • At the server, the hostname value should resolve locally. When I used some outer IPs, Tomcat didn't start up correctly (taking a long long time...)

Solution 3: Use Jolokia and expose some special URLs to the outer world

Jolokia is an agent which translates JMX queries and operations to REST-HTTP/JSON. It's really easy to write a Nagios check script. I did one in Python with took something like an hour.

What I did:
1. Download the WAR from the Jolokia download page.
2. Unzip the WAR to edit web.xml
3. Modified the web.xml, uncommenting the authentication things
4. Zip the war again and drop it into the tomcat/webapps folder
5. Add a user with the "Jolokia" role to conf/tomcat-users.xml
6. Restart Tomcat
7. Test it with a browser at /jolokia/ (The browser should show an authentication dialog.)
8. Search for jolokia nagios plugins or write one.

With a little bit more of time, I modified my Nagios plugin (which I use from Shinken, not Nagios) to display all Heap Memory data into MBytes or percentage, so you can something like this:

./check_jolokia_heap -U http://......  -c 80% -w 90% -u -p

and here is an example output (should be in one line):
JMX OK HeapMemoryUsage.used=439.57{max=1185.5;init=1248.0;used=439.57;committed=1185.5}|HeapMemoryUsage.used=439.57;998.4;1123.2

Note that, although we specify -w and -c arguments in percentage the values are translated into MBytes.

If the -P flag is given the values are translated into percentages:

JMX OK HeapMemoryUsage.used=34.96%{max=1185.5;init=1248.0;used=436.32;committed=1185.5}|HeapMemoryUsage.used=34.96%;80.0%;90.0%

If you're interested, leave a comment.

Solution 4: Invoke a JMX monitoring through SSH

Before we begin, let's talk about the pros and cons:

  • Pros: You don't have to hassle with JMX configurations.
  • Cons:
    • The JMX monitoring command is invoked at the target machine. Make sure you have enough memory
    • If there is any SSH issue, the command will fail, although the JVM may work correctly
    • You need SSH, of course.


Basically, you don't have to bother about JMX ports, firewalls, etc. Just install the monitoring plugin in the target machine and invoke it through SSH.

Now, the question is, how to to invoke it automatically with no direct SSH connection? (Remember that the host is not directly accesible?)

Here you have two solutions:

a) Configure you firewall to forward SSH port to the target machine

b) Use SSH ProxyCommand: Define in the ~/.ssh/config SSH configuration of the monitoring account something like this:

# Our proxied destination host
Host destination-host

  ProxyCommand ssh intermediate-host -W %h:%p

Make sure you can reach the intermediate host without password authentication:

ssh-keygen   #if you don't have already any keypair generated
ssh-copy-id intermediate-host

Now, test you connection to the destination-host:

ssh destination-host

You should get a prompt if you trust the destination certificate's fingerprint and after that the password prompt. If everything works as expected, just copy your public key to the destination host:

ssh-copy-id destination-host

Finally, copy your monitoring plugin to the destination host and invoke it, e.g. in Nagios / Shinken, your command definition could be something like this

define command {
    command_name    check_tomcat_mem_heap
    command_line    $NAGIOSPLUGINSDIR$/check_jmx \
        -U service:jmx:rmi:///jndi/rmi://'$HOSTADDRESS$':'$ARG1$'/jmxrmi \
        -O java.lang:type=Memory -A HeapMemoryUsage \
        -K used -w '$ARG2$' -c '$ARG3$
}

27 may. 2014

Liferay 6 and Sentry

Introduction

Sentry [1] is a great tool for error tracking and Liferay [2] is a very popular portal software that we deploy for out customer as part of our main product.

Log4j configuration with Liferay

As stated in [3], custom log4j configuration is done adding these files:
  • portal-log4j-ext.xml
  • log4j.dtd
to folder: tomcat-[version]/webapps/ROOT/WEB-INF/classes/META-INF where [version] is something like 7.0.23 and depends on the concrete Liferay version you are working with.

The portal-log4j-ext.xml overrides the file portal-log4j.xml which can be found inside portal-impl.jar (in webapps/ROOT/WEB-INF/lib). You can get a copy from here [4]. The companion file log4j.dtd can be found here [5].

Get Sentry Java Client (raven-java)

To log any errors in our Liferay instance to Sentry, we need a Sentry Java Client which works together with log4j (a Log4j appender) and can be downloaded from here [7].

Although Liferay already includes log4j 1.2.x we choose the jar that includes all dependencies (for a reason I explain below):

Download the file raven-log4j-[version]-jar-with-dependencies.jar where [version] is currently: 1.0-SNAPSHOT. 

Configure Liferay to log to Sentry

This is done it two steps:
  1. Copy the downloaded raven-log4j JAR to tomcat-/lib
  2. Customize portal-log4j-ext.xml
Copy raven-log4j file
Copy the downloaded raven-[version]-jar-with-dependencies.jar to tomcat-[version]/lib.
Although all the java dependencies should normally live together with the web application and the downloaded file contains log4j 1.2.x, we have chosen this approach for the following reasons:

  • Then specific log4j appender and Sentry java client (raven-java) will be available for other deployed web applications. Take into account that in Liferay deployment each extra portlet is considered a separate web application with its own J2EE application context. Each of them will have to be configured individually to log to Sentry.
  • The log4j version included in raven-log4j-[version]-jar-with-dependencies.jar has the same major and minor version (1.2), i.e. the Sentry log appender (SentryAppender and AsyncSentryAppender) should be 100% compatible. Special caution should be given when using other versions.
  • The tomcat/lib contains class libraries (JAR) which are shared between all of the deployed web applications (WAR). But (!) the J2EE classloader magic avoids calls from a shared class library back to a web application class. In this case, it doesn't happen.
  • The web application classes and class libraries have preference when searching for a class
Customize portal-log4j-ext.xml
Following this example [6] where Nuxeo is configured for Sentry, we have to add an appender and activate it in portal-log4j-ext.xml, like this:
 <?xml version="1.0"?>  
 <!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">  
 <log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">  
   <appender class="org.apache.log4j.ConsoleAppender" name="CONSOLE">  
     <layout class="org.apache.log4j.PatternLayout">  
       <param name="ConversionPattern" value="%d{ABSOLUTE} %-5p [%c{1}:%L] %m%n" />  
     </layout>  
   </appender>  
   <appender class="org.apache.log4j.rolling.RollingFileAppender" name="FILE">  
     <rollingpolicy class="org.apache.log4j.rolling.TimeBasedRollingPolicy">  
       <param name="FileNamePattern" value="@liferay.home@/logs/liferay.%d{yyyy-MM-dd}.log" />  
     </rollingpolicy>  
     <layout class="org.apache.log4j.PatternLayout">  
       <param name="ConversionPattern" value="%d{ABSOLUTE} %-5p [%c{1}:%L] %m%n" />  
     </layout>  
   </appender>  
   <appender class="net.kencochrane.raven.log4j.SentryAppender" name="Sentry">  
     <param name="dsn" value="http://[two hashes separared by a colon]@log.tangrambpm.es/5" />  
     <filter class="org.apache.log4j.varia.LevelRangeFilter">  
       <param name="levelMin" value="INFO" />  
     </filter>  
   </appender>  
   <category name="com.ecyrd.jspwiki">  
     <priority value="ERROR">  
   </priority></category>  
 ...  
   <root>  
     <priority value="INFO">  
     <appender-ref ref="CONSOLE">  
     <appender-ref ref="FILE">  
     <appender-ref ref="Sentry">  
   </appender-ref></appender-ref></appender-ref></priority></root>  
 </log4j:configuration>  

After theses steps and a Liferay restart you should be done.

References

30 dic. 2013

¿Por qué los polvorones y mantecados contienen E-320?

Pregunta abierta a fabricantes y distribuidores de polvorones y mantecados y especialistas de alimentación


Hasta ayer no sabía nada sobre el antioxidante E-320, pero al leer los ingredientes de los polvorones y mantecados, y en la tónica de revisar los ingredientes de los alimentos para vigilar la alimentación en nuestra familia, me he encontrado con este ingrediente que desconocía.

Según las referencias consultadas E-320 es un antioxidante sintético utilizado en la industria industria petrolífera empleado para conservar grasas y cuyo consumo debería evitarse por la posibilidad de la aparición de los siguientes efectos adversos:

  • Hiperactividad
  • Asma
  • Urticaria
  • Insomnio
  • Aumento del colesterol en la sangre
  • Problemas de metabolismo en el hígado
  • Adormecimiento
  • Tumores cancerígenos

Aunque en Europa y USA esté permitido su uso, en Japón está prohibido.

Por lo cual me gustaría saber:

  • ¿Para qué hace falta añadir este ingrediente? ¿No es la manteca de cerdo y el azúcar suficiente para conservar estos alimentos?
  • ¿Qué alternativas existen a E-320 para el caso de polvorones y mantecados?
  • ¿Por qué no se están usando estas alternativas?

Gracias.

Atentamente,
un ciudadano a pie que le gustaría disfrutar de los dulces navideños sin preocupaciones


Referencias (todas consultadas el día 30/12/2013):

Por internet pueden encontrarse fácilmente otras referencias.

27 dic. 2013

Make Java applets work with chromium-browser on Ubuntu

The problem

On some Ubuntu versions, notably Ubuntu 12.04 LTS and some newer ones, Chromium >= 30 may not execute Java applets. In some browser versions, it will crash the browser process.

Run your own test:


The result should be something like this:



The problem is a name clash with a Java library: libnet.so.

This does not happen with all Chromium versions. In recent versions, all the provided libraries can't be linked statically into one single executable as before. (The resulting binary would be too big.) Hence the libs directory.

The solution

Of course, you can use Mozilla Firefox or some other browser which allows the use of Java applets. For me, the Google Chrome browser also works correctly.

So if you still want to use Chromium, go on.

Let's change all libnet occurrences with some other unused library name.

Here we go:

sudo su
apt-get install bbe
cp -a /usr/lib/chromium-browser /usr/lib/chromium-browser.bak
cd /usr/lib/chromium-browser
for f in `rgrep -l libnet .`
do
  bbe -e 's/libnet/libxet/' $f > $f.new
  mv -f $f.new $f
done
mv libs/libnet.so libs/libxet.so
mv libs/libnet_with_v8.so libs/libxet_with_v8.so
chmod +x chromium-browser

Before running the test again make sure all chromium-browser processes have stopped and kill them if necessary:
ps aux | grep chromium-browser
killall chromium-browser

Now run the applet test again and see if it works.

Take into account that the next package update may overwrite these changes.

Conclusion

Although a bit hackish, if you have to use Java applets in daily work like me and you really want to use Chromium, here you have a solution.

3 dic. 2013

Buildout errors with gocept.recipe.env

The syntom

I get a really very strange error when running buildout.

While:
  Installing nginx.
  Getting option config:PS1.
Error: The option name in substitution, ${debian_chroot:+($debian_chroot)},
has invalid characters.
One strange behavior is that it does not occur every time. I still haven't discovered exactly when this error occurs. But the most strange and surprising fact is that it corrupts the buildout control file .installed.cfg, making it unusable for later buildout runs.

The problem

After digging a while I have discovered that my buildout (incepted by co-workers) is using the gocept.recipe.env 1.0 to get a environment variable (USER), and the error is related to the PS1 environment variable. It includes a dollar sign ($) and or this recipe or buildout (2.1.0) is not escaping it properly.

The solution

I opted to separate my part into two and to use collective.recipe.environment instead.

[mypart]
recipe = gocept.recipe.env
key = value
...

is converted to:

[enviroment]
recipe = collective.recipe.environment

[mypart]
USER = ${environment:USER}
key = value
...

I hope that this solution will be less erroneous / more solid.

8 nov. 2013

Enable Wifi Tethering on my MTK6589 Flying F600 with Android Jelly Bean 4.1.2

The problem

Google has introduced changes in the tethering module since Android 4.1.2 to allow carriers to charge their customers Internet usage from other devices. [1]

On the other hand, I have a less famous Android 4.1.2 mobile phone, the Flying F600, which uses the Mediatek MTK6589 Quad Core processor, so I needed a rooting kit that works with this phone and preferably with my MacBook Pro with OS X 10.8.5.


The solution

Overview

  1. Download and root the phone
  2. Create a script for enabling NAT using iptables
  3. Download and install Script Manager
  4. Configure the script to execute automatically
  5. Add the tethering widget to your home screen.

Steps

1. Download and root the phone

Before I did this I had no idea, but I'm a hacking guide, so I have plenty of knowledge SSH, shell scripting, networking, etc.

The root kit which I found to work well is the one I found at droidchina forums [2]. The download link is here. Previous to downloading I had to register.

I just followed the steps of the article. The compressed zip file includes the Android Debug Bridge executable ADB for Windows, Linux y OS X.

Before running the rooting command (run.sh), make sure your phone's USB connection is debug mode. In the following screen shot you can see the spanish version.


What I found interesting is the "adb shell" command which lets you hack directly your device. In combination with "su -" you can do a lot of things. It's just a custom Linux OS:

2. Create the script for enabling NAT

After some attempts to enable NAT using iptables, I found that the following script worked best:
su -c "iptables -tnat -F"
su -c "iptables -tnat -A POSTROUTING -s 192.168.0.0/16 -o ccmni0 -j MASQUERADE"
su -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
You can write the script using "adb shell" and "vi". Save the script to /sdcard. I called it "tether_enable". You could also write it using an Android Text Editor like Jota or using a Terminal with vi or some other native commands like cat, etc.

Some explanations here:

  • The first line flushes iptables "nat" routing table. This is good when the script is executed several times.
  • The "ccmni0" interface is my normal public 3G data services interface. You can use the "netcfg" command to find out yours. Anyway, you can try it without specifying the "-o ccmni0" option of iptables.
  • I tried it with "-A natctrl_nat_POSTROUTING" found in several forums but it didn't work for me. I still don't know why. So I used the POSTROUTING chain.
  • Whenever I enable or disable tethering, ip_forward is reset to 0.

3. Download and install Script Manager

Script Manager (SManager) is an excellent application to enable script execution. It allows you to execute scripts directly, mark it as favorite, execute on boot or on network change.



4. Configure the script to execute automatically

After installing it, I selected my /sdcard/tether_enable script and marked it as favorite, root and net. This executes the script whenever the network config changes.

       Select the script       
Mark as favorite,
root and net

5. Add the tethering widget to your home screen

Locate the tethering widget
Hold and drag it to
 the home screen       
Choose Wifi tether      
Widget is on home screen

Conclusions / What I learned

  • Android is just a custom Linux
  • Rooting your Android phone opens lots of new possibilities
  • Rooting is easy
  • Tethering is fixed by enabling NAT (with iptables)
  • How to capture screens: use "adb 

References