blog

New AWS CloudFormation YAML syntax and variable substitution in action

I’ve been using CloudFormation YAML syntax for a while now with Ansible and the serverless framework which would convert the YAML to JSON before uploading the template. That already gave me the YAML advantages of e.g. code comments, not having to care about commas etc.

A few days ago, AWS announced native YAML support for CloudFormation templates, in addition to the existing JSON format.

And along with that they added new shorthand syntax for several functions

Let’s go through a template which I created not only in order to get used to the new syntax :)

mehr lesen 1 Kommentare

How to install and use a newer version (3.x) of NPM on AWS Lambda.

My current experiment is to build a serverless deploy pipeline (With AWS CodePipeline) which uses AWS Lambda for the build steps. One step includes to invoke NPM to build a static website out of JavaScript components (which would be deployed to an S3 bucket in a later step).

Ok, so let's go ahead and look what is actually installed in the preconfigured Node 4.3 env on AWS Lambda. First we want to find out if NPM is actually already installed. So we just create a new Lambda function which invokes a `find’ command, here is the used source code: 

exports.handler = (event, context, callback) => {
var child_process = require('child_process'); console.log(child_process.execSync('find /usr -name npm -type f', {encoding: 'utf-8'}));
};

And, voila, we found something, here is the output: 

/usr/local/lib64/node-v4.3.x/lib/node_modules/npm/bin/npm

So let's try to execute it! 

console.log(child_process.execSync('/usr/local/lib64/node-v4.3.x/lib/node_modules/npm/bin/npm version', {encoding: 'utf-8'}));

And here is the output:

module.js:327
    throw err;
    ^

Error: Cannot find module '/usr/local/lib64/node-v4.3.x/lib/node_modules/npm/bin/node_modules/npm/bin/npm-cli.js'
    at Function.Module._resolveFilename (module.js:325:15)
    at Function.Module._load (module.js:276:25)
    at Function.Module.runMain (module.js:441:10)
    at startup (node.js:134:18)
    at node.js:962:3

    at checkExecSyncError (child_process.js:464:13)
    at Object.execSync (child_process.js:504:13)
    at exports.handler (/var/task/index.js:4:29)

Ok, that doesn’t look good, does it? Actually the ‘node_modules/npm/bin/node_modules/npm/bin/npm-cli.js’ part looks broken.

 

Ok, so my next step was to find the correct path to npm-cli.js, so I have a chance to call it without the apparently broken executable wrapper: 

console.log(child_process.execSync('find /usr -type f -name npm-cli.js', {encoding: 'utf-8'}));
/usr/local/lib64/node-v4.3.x/lib/node_modules/npm/bin/npm-cli.js

So let’s try to call it directly:

console.log(child_process.execSync('node /usr/local/lib64/node-v4.3.x/lib/node_modules/npm/bin/npm-cli.js version', {encoding: 'utf-8'}));

gives us:

{ npm: '2.14.12',  ... }

Yay! We got NPM working!

 

But NAY, it’s an old version! 

 

So let's go ahead and try to install a newer version! Lambda gives us a writable /tmp, so we could use that as a target dir. NPM actually wants to do much stuff in the $HOME directory (e.g. trying to create cache dirs), but it is not writable within a Lambda env.

So my “hack” was to set the $HOME to /tmp, and then install a newer version of NPM into it (by using the --prefix option):

process.env.HOME = '/tmp';
console.log(child_process.execSync('node /usr/local/lib64/node-v4.3.x/lib/node_modules/npm/bin/npm-cli.js install npm --prefix=/tmp --progress=false', {encoding: 'utf-8'}));
console.log(child_process.execSync('node /tmp/node_modules/npm/bin/npm-cli.js version', {encoding: 'utf-8'}));

Ok, NPM got installed and is ready to use!

npm@3.10.5 ../../tmp/node_modules/npm

The last step is to symlink the npm wrapperso it can be used without hassle. And actually many build systems seem to expect a working npm executable:

fs.mkdirSync('/tmp/bin');
fs.symlinkSync('/tmp/node_modules/npm/bin/npm-cli.js', '/tmp/bin/npm');
process.env.PATH = '/tmp/bin:' + process.env.PATH;
console.log(child_process.execSync('npm version', {encoding: 'utf-8'}));

 And here we go!  Now it’s possible to use a up-to-date version of NPM within a Lambda function. 

 

Some additional learnings:

  • NPM needs a lot of memory, so I configured the Lambda function with max memory of 1500MB RAM. Otherwise it seems to misbehave or garbage collect a lot.
  • You should start with a clean tmp before installing NPM in order to avoid side effects, as containers might get reused by Lambda. That step did it for me:
    child_process.execSync('rm -fr /tmp/.npm');
    // ... npm install steps ...
  • Downloading and installing NPM every time the build step is executed makes it more flaky (remember the fallacies of networking!). It also reduces the available execution time by 10 seconds (the time it takes to download and install NPM). One could pack the installed npm version as an own Lambda function in order to decouple it. But that’s a topic for another blog post. 
0 Kommentare

Keeping your Pocket list clean with pocketcleaner and AWS Lambda

Over the last years my Pocket reading queue got longer and longer. It actually dated back to stuff from 2013. Over the time a realized I would never ever be able to keep up with it again.

 

Some days ago I found out that Daniel (mrtazz) developed a nice tool named pocketcleaner which archives too old Pocket entries. I thought "Hey great, that's one solution to my problem, but how to execute it?". People who know me might already have an idea :) I don't like servers in terms infrastructure that I have to maintain. So I thought AWS Lambda to the rescue!

 

And here it is: An Ansible playbook which setups a Lambda function which downloads, configures and executes the Go binary. It can be triggered by a AWS event timer. No servers, just a few cents per month (maximum!) for AWS traffic and Lambda execution costs.

0 Kommentare

The PHP Equality Table

Inspired by the "JavaScript Equality Table" I created the same table for PHP:

mehr lesen 0 Kommentare

devopsdays Ghent recap

!!! ATTENTION: Highly unstructured braindump content !!!

mehr lesen 0 Kommentare

External MySQL slaves with RDS reloaded

In an earlier first post I demonstrated a way to connect an external slave to a running RDS instance. Later then AWS added the native possibility to import and export via replication.


In my case, several problems popped up:

  • My initial blog post did not show how to start from an existing data set, e. g. do a mysqldump, and import
  • RDS does not allow --master-data mysqldumps as “FLUSH TABLES WITH READ LOCK” is forbidden in RDS
  • So we do not have any chance to get the exact starting point for reading the binlog on the slave.


mehr lesen 2 Kommentare

PuppetConf 2013 recap

Better late then never. Here’s my little puppetconf 2013 recap.

mehr lesen 0 Kommentare

Diploma thesis

Disclaimer: German.

 

Auch Aaron Swartz zu Ehren - veröffentliche ich meine Diplomarbeit "Wissensmanagement als integraler Bestandteil des Software Engineerings - Konzeption einer Vorgehensweise unter Einbindung agiler Modelle am Beispiel des Web 2.0-Unternehmens Jimdo".

 

In der Diplomarbeit wird untersucht, inwiefern die Anwendung agiler und Lean- Softwarentwicklungsmethoden wie Scrum, Software Kanban, Extreme programming und DevOps schon Wissensmanagement implizit "betreiben".

Hierzu werden die einzelnen Methodenbausteine (z. B. "Daily Standup" oder "Pair programming") vor dem Hintergrund des Wissensmanagement als Geschäftsprozess untersucht. Daraus wird ein Wissensmanagementmodell entwickelt, welches sich am Unternehmen Jimdo ausrichtet.

 

Vielen Dank an meine Frau Mila, Boris, Judith und Bracki, die fleißig und unermüdlich korrekturgelesen sowie wichtige inhaltliche Kritik geliefert haben!

mehr lesen 1 Kommentare

puppet-rspec debugging

While introducing rspec-puppet into a big and grown puppet codebase at Jimdo we needed to debug stuff and get more verbose output while writing the first tests. As the interwebs aren't very chatty about the topic, here for all the distressed googlers:

mehr lesen 2 Kommentare

devopsdays in hamburg

mehr lesen 0 Kommentare

mysql prompt tuning

mysql> prompt (\u@\h) [\d]>
PROMPT set to '(\u@\h) [\d]>'
(debian-sys-maint@localhost) [cms]>

persistent in bashrc:

 

export MYSQL_PS1='(\u@\h) [\d]> '

 

0 Kommentare

rdiff-backup: the easiest way to backup (and restore)

Today I finally managed to set up an incremental backup for my workstation. What do we need? Well, nothing more than rdiff-backup, an opensource command line tool with all the powers you need.

 

it just backups. and your last backup is always available 1:1 at the destination (no strange storage formats etc., just dirs and files). Diffs and metadata are stored separately. So if you want a backup that does its job, is plain, is easy to restore, has no unneccessary features and "just works" than rdiff-backup is the right tool for you.

 

Here a small bash script I write to accomplish the mission:

#!/bin/bash
P=/home/soenke
DESTINATION=mydestionationserver.local
INCLUDE="
$P/Documents
$P/.gnupg
"

echo "$INCLUDE" | rdiff-backup --include-filelist-stdin --exclude $P/ $P/ $DESTINATION::/home/soenke/backup
mehr lesen 0 Kommentare

Integrating Google SSL Search into your Firefox Searchbar

Google search is now also possible via SSL. If you are paranoid like me you might wanna have it enabled by default - while there are several firefox plugins around there, here's how to do it on your own:

 

  1. Copy an existing Google Search Plugin XML into your Profile under "searchplugins" (maybe the diectory has to be created first). In Ubuntu for example it's placed /usr/lib/firefox-addons/searchplugins/<your locale>/google.xml
    Your own Firefox Profile is located in $HOME/.mozilla/firefox/<Profile Name/ (Ubuntu).
  2. Edit the freshly copied google.xml within your profile, change the ShortName to e. g. "My SSL Google" and <Url type="text/html" method="GET" template="http://www.google.com/search"to
    <Url type="text/html" method="GET" template="https://www.google.com/search">
  3. Restart Firefox and switch to your new own Goodle Search Profile (the scrolldown thingy with wikipedia, google, yahoo etc.)

This tutorial is also valid for other OSes, but you may have to find the appropiate locations / directories yourself ;)

 

I've been using my own google search xml for ages anyway to change the default language to english because it's annoying to have auto-geo-autodetected language search results everytime esp. if you search for technical things most of your time (Yes you could also set your preferred language in google, stay logged in all the time, let your cookies survive browser restarts etc, but that's no option for me). To accomplish this, just add a param line to the xml file like this: <Param name="hl" value="en"/>

0 Kommentare

PHP: SortingIterator

I just had the need for an Iterator that can sort itself by a user defined callback.

 

My special use case is that the DirectoryIterator of PHP does not sort the file list so it's pretty random. But my program logic relies on files being sorted by filename.

 

So here's the little class:

mehr lesen 1 Kommentare