Ibuildings - Web & Mobile App Development

The problem with Apache’s approach to dealing with multiple clients, is that there’s only ever a limited amount of Client processes available. This is usually is around a few hundred on common webservers.

Because of this, it becomes necessary to handle HTTP requests as quickly as possible. As soon as a request is handled, it can go on serving the next. If a client happens to have a slow connection, this can have a direct effect on the scalability of your frontend server.

A common way to fight this, is to put a caching server in front of your webserver, such as Varnish or Squid. These webservers are better suited to deal with many clients. This will allow your Apache server to send back HTTP responses quickly to the reverse proxy, and let the proxy deal with sending back the response to the client.

However, this doesn’t deal with slow requests. Generally, these proxy servers will open connections directly to the backend webserver to avoid having to buffer larger request bodies.

Because PHP installations generally use apache ‘prefork mpm’, the number of possible connections is considerably low. This is also often the case with Fast-CGI based webservers, such as nginx and lighttpd. So if you were to just able to open up a few hundred connections, and drip in the bytes for the request body it would be very easy to take these servers down.

To test this theory, I wrote a simple python script that does exactly this, you can grab it from github. To use it, try something like this:

python slowdeath.py --threads 200 http://localhost/

In my case my webserver was limited to 150 connections. It took about a second for it to stop serving requests.

Big warning: This tool is for research purposes only. Use at your own risk, and only on servers you own.

To take out a server, simply specify a number of threads higher than the MaxClients or whatever setting your webserver happens to use. Note that I only tested this on a few servers, so results may vary. Side effects include diarrhea, rashes, blackouts and death. Do not use while driving.

The interesting people you meet at user group meetings, all the stuff you learn at conferences and the freely available open source projects that are out there. Those are just three of many things that make the PHP community into something awesome. However, every once in a while I hear some plea for the great PHP community that makes me a bit nauseous. “Sharing code, knowledge, elePHPants, it’s all so fun to share and be part of this great club of lovely people! I share my project with you and you share your project with me. And then we hug! Weeeee!!!”

Photo by Nelson Piedra

Well f*ck that. Of course, whenever you regularly meet a group of people you will make some new friends, especially when you share a common interest. You might enjoy drinking beers with them or do some coding on their projects, because you think it’s a great project and you want to help them out. However, you will also make friends when you go out to the pub. Hell, you will probably even make friends at the weekly meeting for people with ingrown toenails – which is great, sure! But saying ingrown toenails are a great thing because “you meet such great people” just seems a bit awkward to me.

Don’t get me wrong: I love the sharing, partying and beer drinking as well, and I wouldn’t want to miss it. The most interesting people you meet on conferences are the people you meet on the social events afterwards, but it’s all a consequence, not a cause. Saying you are part of the community because of it, is like saying to your date that you went through the whole process of “having sex and all” because you like smoking the cigarette at the end so much.

So besides the hippy arguments, the PHP community also has some great, more down-to-earth things to offer for companies, developers and everybody else involved. Below I just listed a couple of arguments I could come up with by brainstorming for ten minutes. I’m pretty sure I might have missed one or two, but it should at least give you an idea.

Contributing to open source projects
On first sight, contributing to open source projects might look like charity. You and your noble steed have come to save the day, and with all your goodness you fix a bug in the project. Hooray! But contributing has some other upsides as well. First of all, it gives you a bit more renown and might make your resume look more interesting. How great would it be if you could put “regular contributor to the Zend Framework project” on there? By contributing you would probably learn a lot as well, just by looking into other people’s code and writing documentation or tests for it.

Or if you’re using an open source product but you need to fix or extend it for your own purpose, giving back the changes you made means you can now keep updating to newer versions of the product. Your changes will be in it, so you won’t be stuck with the same version for the rest of your life because you added some custom hacks and an update would overwrite those changes.

Open sourcing your own projects
This one is rather simple; open sourcing your own project means that – if you do it right – people will start contributing to it. When your company has developed a somewhat useful tool, open sourcing it will probably mean that other people will start using the tool as well. Eventually, some people might come and fix some bugs in it, improve the tool and build some new features for it. Your tool gets enhanced, extended, better – for free!

Conferences and user group meetings

Events like conferences are a great place to learn and to get inspired, but can also be used for recruitment and networking. People might learn from the talks, or from talking to each other. Talks can also be inspirational and work as a trigger for somebody to start investigating a technique or a tool he didn’t think of before.

If you do it right, speaking at such events will give you and the company you’re representing some great renown. Tens, maybe hundreds of potential colleagues, clients and people who might hire you see that you, and the company you’re representing, really know what it’s all about. As a company it’s a great way to show that you take software development seriously, and that your office is a great place to work. The speaker gets a bit more fame, and can add another interesting line on his resume.

I did a little research on this subject by starting a poll about a week ago. The results were somewhat surprising, and they are probably hopelessly unreliable because of the group that answered the poll Still, it gives you a bit of an idea:

67% goes there to talk to interesting people

58% goes there for the talks (which I had expected to be more, since that’s the main thing conferences have to offer – the stuff they’re selling)

53% goes there for the hugging and beer drinking

25% goes there to speak (you might see now what I meant by hopelessly unreliable )

24% goes there to promote themselves

16% goes there to promote their company, brand or product

8% goes there to recruit new developers

19% selected “other”. Answers included “I go there to convert all over to ruby” and one voter goes to PHP conferences to “be away from the missus”

The community online
Whether it’s on IRC or on some forum, there are some great resources out there for getting and sharing information, pretty much for the same reasons as the speaking and attending talks part: the person answering a question get a bit more renown, the person asking the question has his problem solved and can continue. Also, answering questions often forces you to do a little research yourself, how did this work again exactly? Somebody else might give an alternative answer to the question that would work just as well, but you hadn’t thought about that solution yet. You can really learn a lot from answering other people’s questions!

The hippy thing
And then of course, there’s the hippy thing. A great network of people that help each other out, give each other advice, do an open source project together and then drink some beer. This post is to show that it’s not the only thing that matters – but of course it is an important reason why it’s a lot of fun to be a part of the PHP community.

Since may 11 TLD’s (top-level domainnames) have been added. In order for this to work successfully, a lot of applications will have to be fixed.

Many email-validation scripts might use an approach like this:

$ok = preg_match('/^[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,6}$/i', $email);

This one is pretty simple, it matches the most common address formats, as long as the tld (.com, nl, .uk, etc) is under 6 characters. For a bit more sophistication you might want to ensure that the tld is a bit more valid:

$ok = preg_match('/^[A-Z0-9._%+-]+@[A-Z0-9.-]+\.(?:[A-Z]{2}|com|org|net|edu|gov|mil|biz|info|mobi|name|aero|asia|jobs|museum)$/i',$email);

Note: both these regexes were taken from regular-expression.info. The top google hit, and decent examples.

The new TLD’s use non-ascii characters, and they might become aliases for existing top-level domains, or new tld’s altogether. Here are the currently working examples:

• http://مثال.إختبار – Arabic.
• http://例子.测试 – Chinese (simplified)
• http://例子.測試 – Chinese (traditional)
• http://παράδειγμα.δοκιμή – greek
• http://उदाहरण.परीक्षा Hindi
• http://例え.テスト – Japanese
• http://실례.테스트 – Korean
• http://مثال.آزمایشی – Persian
• http://пример.испытание – Russian

At first sight these look like regular utf-8, characters, but if you look at the sourcecode of this page, you’ll notice that it’s actually encoded differently.

The korean url http://실례.테스트, is actually encoded as http://xn--9n2bp8q.xn--9t4b11yi5a/. This is called Punycode.

If you want support for these new urls (and thus domainnames in emails), you should have support for punycode. You will likely receive UTF-8 encoded domainnames for email address (example@실례.테스트), but internally you must make sure that you only deal with the punycode representation.

This translating is also what modern browsers do. If you were to paste “http://xn--9n2bp8q.xn--9t4b11yi5a/” directly in the firefox address bar, it will show you the UTF-8 characters instead. Firefox will re-encode to punycode though and use that format for HTTP requests.

The best way really to check for valid email addresses is to use a very liberal regex, but verify with a simple MX record lookup if a mailserver exists for the given domain. This example is an expansion on the first regex.

$email = 'example@xn--9n2bp8q.xn--9t4b11yi5a';
 
if(preg_match('/^[A-Z0-9._%+-]+@([A-Z0-9.-]+\.[A-Z0-9-]{2,})$/i', $email,$matches)) {
 $hostname = $matches[1];
 if (!getmxrr($hostname, $hosts)) {
 echo "Host has an MX record\n";
 } else {
 echo "Host does not exist or does not have an MX record\n";
 }
} else {
 echo "Email address did not match regular expression\n";
}

The preceeding code does not convert UTF-8 to punycode though. There’s not yet an easy native way in PHP to do this, but Pear’s Net_IDNA2 provides a way. The implementation seems very complex though, and leaves me wondering if there’s an easier way to go about it.

Although I’ve been using NetBeans as an IDE for a couple of months now, I’ve just recently found out about the nifty PHPUnit integration that comes with it. Once you’ve told NetBeans where your tests are, where PHPUnit is and how PHPUnit should be called to run your tests, you can run the unittests and see the test results in the IDE!

Step 1: tell NetBeans where your tests are

Open the project preferences and enter the directory containing your unittests in the field ‘Test folder’.

If you did this right, you will notice that NetBeans doesn’t show this folder anymore in the projects tree, but instead it shows a separate tree: underneath the tree “Source files” there’s now also a tree called “Test files”:

Step 2: tell NetBeans where PHPUnit is

Go to Netbeans’ general preferences, open the tab ‘PHP, and then the tab ‘Unit Testing’. Here, enter the path to PHPUnit:

Step 3: tell NetBeans how to run the tests

Finally, we need to tell NetBeans how to run our tests. This can be done by pointing it to the phpunit.xml, in the Project Properties dialog (PHPUnit menu):

That’s it! You can now run your unittests using NetBeans!

Running unittests

In the ‘run’ menu a new option has appeared. Just underneath ‘Run Project’ option there’s a new option, ‘Test Project’. When you select this option NetBeans will run your unit tests and show the results:

As you can see something went wrong. It will not surprise you that clicking on the failed test will open the right unittest in your editor, showing you the line that failed so you can start figuring out what went wrong. The Test Results tab is sometimes a bit cryptic as to what went wrong, so it might help to open the Output dialog as well (menu Window -> Output -> Output), as the raw output of phpunit is captured here.

Code coverage

NetBeans can also capture the code coverage while you’re testing. To do this you first need to enable this by right-clicking the project and then selecting “Code Coverage” -> “Collect and Display Code Coverage”. Please note that you have to got the xdebug extension installed before you can do this, otherwise PHPUnit won’t run.

Once enabled you can run phpunit again just as you did earlier, but when it’s done running it will show in your source files which code was tested and which code was not, by marking it either green or red:

Upsides

There are probably a lot more things you can do with the NetBeans – PHPUnit integration. I haven’t got time yet to investigate everything that’s possible, but I think that this already is quite nifty. Opening my iTerm and typing ‘phpunit’ usually isn’t that much of a hassle, but generating, opening and browsing the code coverage HTML can sometimes be quite time consuming. Now, I can just press ^ F6 and the tests are run and the tested/untested lines are marked green and red in my editor. Great!

Downsides

There are downsides. First of all, you need to be able to run PHPUnit from the OS your IDE is on – so probably that means that you need to be running your webserver locally. Personally I like to run my webserver on a vmware image that represents the production environment, and I share the folder where my project is in using a hgfs mount. This means I couldn’t use PHPUnit that I installed on my vmware image, but I had to install it again on Mac OSX. This not only means extra work, but it also means the environment I’m running my tests on it not longer representative to the production environment.

Before updating your production environment, I think it’s a good idea to first run your tests once more on a more representative environment. Nevertheless the NetBeans – PHPUnit integration saves me quite some time when writing tests, because basically everything is happening in the same screen.

I just released version 1.3.0 of SabreDAV. Uptake has been very strong, especially for the CalDAV components. The biggest change is a big performance boost for most tree operations.

To upgrade, download the new file here, or if you installed it using pear:

pear upgrade sabredav/Sabre_DAV
pear upgrade sabredav/Sabre_CalDAV

To install using pear:

pear channel-discover pear.sabredav.org
pear install sabredav/Sabre_DAV
pear install sabredav/Sabre_CalDAV

There is a list of 4 (smallish) backwards compatibility breaks in the API. You can read about it in the migration guide.

Full list of changes:

• Added: Cache layer in the ObjectTree.
• Added: childExists method to Sabre_DAV_ICollection. This is an api break, so if you implement Sabre_DAV_ICollection directly, add the method.
• Changed: Almost all HTTP method implementations now take a uri argument, including events. This allows for internal rerouting of certain calls. If you have custom plugins, make sure they use this argument. If they don’t, they will likely still work, but it might get in the way of future changes.
• Changed: All getETag methods MUST now surround the etag with double-quotes. This was a mistake made in all previous SabreDAV versions. If you don’t do this, any If-Match, If-None-Match and If: headers using Etags will work incorrectly. (Issue 85).
• Added: Sabre_DAV_Auth_Backend_AbstractBasic class, which can be used to easily implement basic authentication.
• Removed: Sabre_DAV_PermissionDenied class. Use Sabre_DAV_Forbidden instead.
• Removed: Sabre_DAV_IDirectory interface, use Sabre_DAV_ICollection instead.
• Added: Browser plugin now uses {DAV:}displayname if this property is available.
• Added: Tree classes now have a delete and getChildren method.
• Fixed: If-Modified-Since and If-Unmodified-Since would be incorrect if the date is an exact match.
• Fixed: Support for multiple ETags in If-Match and If-None-Match headers.
• Fixed: Improved baseUrl handling.
• Fixed: Issue 67: Non-seekable stream support in ::put()/::get().
• Fixed: Issue 65: Invalid dates are now ignored.
• Updated: Refactoring in Sabre_CalDAV to make everything a bit more ledgable.
• Fixed: Issue 88, Issue 89: Fixed compatibility for running SabreDAV on Windows.
• Fixed: Issue 86: Fixed Content-Range top-boundary from ‘file size’ to ‘file size’-1.

I plan to fully keep supporting the 1.2.* branch, but I’ll backport bugfixes strictly on an on-demand basis. So far there’s been relatively little people stuck on older versions, so I’m only spending time on it in case anyone depends on it.

Thanks to all the people reporting bugs and posting patches!

Along with the release of 10.10, Ubuntu came with a new self-named font. I love it. It’s quirky, yet very legible.

The font is open-source, with a pretty straightforward license, which comes down to: ‘include this license when redistributing. There’s very little good free fonts out there that actually allow you to embed it on your site, but with this one you can.

You can download the ttf’s from here. Embedding it using css is easy:

@font-face {
 font-family: "Ubuntu Sans";
 src: url('font/ubuntu/Ubuntu-R.ttf');
} 
@font-face {
 font-family: "Ubuntu Sans"; 
 src: url('font/ubuntu/Ubuntu-B.ttf'); 
 font-weight: bold
} 
@font-face {
 font-family: "Ubuntu Sans"; 
 src: url('font/ubuntu/Ubuntu-I.ttf'); 
 font-style: italic 
} 
@font-face {
 font-family: "Ubuntu Sans"; 
 src: url('font/ubuntu/Ubuntu-BI.ttf'); 
 font-style: italic; font-weight: bold 
}

This looked immediately brilliant on Firefox, but Safari acts a bit weird, only anti-aliasing some of the text after hovering over.

Be aware though, this will add about 1.3MB to your page. If you don’t need some of the italic or bold variations, i’d recommend leaving them out.

On font and copyrights

On a more serious note, many people don’t know that most fonts you buy for your websites are never allowed to be straight-embedded into webpages. I’ve seen a number of people embedding their fonts with either @font-face or the dirty (but impressive) cufon, or the worst of all worlds: sifr.

Technically, with any of these technologies you are not just using, but redistributing the font. When you buy a font you are basically only allowed to generate static images. This might not be a big deal for your personal site, but it’s not a wise thing to do for commercial sites.

One feature telnet has and I always missed from ssh was the ^] shortcut, giving you a way to terminate the connection.

ssh has a similar feature. If you setup ‘escape characters’, you can terminate the connection by typing ‘~.’ Just add the following to your .ssh/config:

You can change the character here too, but ~ is the default and a sensible one.

If you’re dealing with crappy ssh connections that often terminate, you can add the following to make the client send a keep-alive package every 60 seconds:

Samy, famous for his worm, released evercookie this week. Evercookie stores cookies is various storage mechanisms such as Flash Local Shared Objects (also known as flookies), HTML5 storage mechanisms and even in the history and cache. When any of these are wiped by the user the script will repopulate it, making it very hard to get rid of your cookies.

This is technique is common to circumvent a users’ privacy wishes, which Clearspring recently got sued for, but it’s put in overdrive.

One good use for it is banning users. In the past I’ve used ips + cookies to ensure a user stays banned, but it doesn’t take much to change your ip address and clear your cookies. All these techniques together make it a lot harder to get through. Because Flash stores it’s flookies in a central place in the operating system, the cookies often even live in multiple browsers and private browsing sessions.

Most of all, I think the tool is made to make a point. It’s very hard for the average user to clear all the tracking information. It should be doable with a press of a button, without losing all your settings and history for every other site.

Since a 2 weeks I’m now employed by IBuildings. First a couple of weeks from their office in Vlissingen, and then if all goes well, to Utrecht.

IBuildings is actually a company I’ve been wanting to work for for a while, so I’m pretty happy. So far it’s a bit of an adjustment to work regular hours again, but I’m having fun. It’s good to be working in an office again. Working from home can definitely get to you after a while. Having lots of talented people around is a big plus.

And: if you know a good place to live in Utrecht, drop me a line! I’m looking to rent a place not too far from downtown

I blogged about Content Security Policy about 2 year ago when it was still called ‘Site Security Policy’. It started as a specification and an add-on, and turned into a patch a bit later. Finally it made it into Firefox 4 beta 1. I think CSP is the next web security revolution, so make yourself aware of how it works and the implications.

So what is it? The short version is that it’s a very effective measure against cross-site scripting. By specifying a policy through the ‘X-Content-Security-Policy’, you can specify exactly from which locations you accept javascript and other content. This allows you to block scripts from any domains unknown to you, and inline scripts altogether.

A simple example

A simple PHP example to see this in action:

If the above code is opened in Firefox 4.0 beta1, the script will not execute, and a warning is added to the “Error Console” (in the Tools menu).

Not only does this header block inline scripts, it also blocks the following:

• eval(). This important for people using eval() to parse json responses.
• setTimeout and setInterval if the function is provided as a string.
• javascript: urls
• HTML event attributes (onclick, onload, etc.).
• All images, plugin objects (flash, quicktime etc.), audio, video, html frames and fonts not served from the same domain as the html page.
• XMLHttpRequest to domains other than the source domain.

Fortunately there are fine grained controls about what you want to allow from which domains. Here are some examples from the specification.

This example starts with “allow ‘self’”, allowing only content from the same domain. The “img-src *” rule allows images from any domain. “object-src: media1.com media2.com” allows <object> tags to use files from media1.com, media1.com and the same domain as the html was served from. To learn more about these, I would recommend just taking a good look at the directives list in the specification.

Options and reporting

Using the ‘options’ directive it’s possible to turn on specific measures. Valid values for options are ‘eval-script’ and ‘inline-script’.

The preceding example allows inline scripts (using html event attributes, or the script tag) as well as the ‘eval()’ function. In general I would try to avoid this though.

When a security rule is violated, it’s possible to get the browser to send a report back to the server. For example, if an image is referenced from a blocked domain, the browser can send a simple report to a url you specify.

This allows you to detect any problems with your policy, or successful attempts by your evil users to inject code. An example of such a report is the following:

Final notes

Using CSP does not mean you can go easy on other security measures. At the moment a very limited amount of users will have support for CSP, so everybody else still needs to be protected. However, it’s still a great idea to implement. Your Firefox users will automatically be protected better, and because of the reporting functionality, they automatically help you detect holes which benefits everybody.

My guess is that CSP is going to be very important, and is here to stay. There are two things you can do to prepare for the future:

1. Figure out your policy. It’s a good idea for your web application to know anyway where resources are coming from. Especially advertisers tend to be bad at using many different domains and scripts using other scripts.
2. Try to avoid any inline scripting, html event handlers and eval(). They are all avoidable, and in my opinion it is a good idea to keep your javascript out of html anyway. This is a big one, because both inline scripts and html events are still very popular. With the popularity of libraries such as jQuery, I do think it will be easier to just grab most of the inline scripts and move them to an external script.