Sylvain Wallez

Google now crawling forms

2008-05-03T23:03:26Z

A few days ago, I started receiving Google alerts about me (yes, I'm following what's said about me on the web) that were linking to search results pages on my own blog, with strange query terms such as "steve" or "near", "idea" or "known".

Why would such searches show up in Google? Who has linked from his website to the search results for these weird words? I tried to find these pages, but failed.

Today I finally found the answer: Google is now crawling through forms, by filling inputs with words they find on the page containing the form. The intent is to crawl the "deep web" that is not normally accessible through regular links.

I'm not sure of the efficiency of this new feature, since every web site that has some sort of product catalog that seems to be only available to humans through forms has semi-hidden link pages to feed the search engines. But it will certainly bring to the surface lots of information that people thought were more or less hidden (not to say protected) behind a form.

Give a box full of keys to a monkey, and it will probably find one that opens your door if you have a weak lock! Inspecting my server logs shows that the current form crawler is not really different from a monkey, trying lots of very common words, 100 per day.

Now the nice thing is that, according to Google's explanation, my blog is a "high-quality site" :-)

Speeding up mobile web applications

2008-04-29T22:54:29Z

It's been nearly one month since I started full speed at Goojet, and I already learned a lot on the mobile web and J2ME.

One of the key issues in the mobile web is latency: connections are long to establish, and data transfer rates are low. Not everybody has a 3G phone nor an unlimited data plan.

Connection establishment latency can be mitigated in two ways:

Use persistent HTTP/1.1 connections. By reusing the same TCP socket for several HTTP requests, you save the non-negligible connection setup overhead. To achieve this, every request and response must carry a Content-Length header.
Open several connections in parallel when you have several items to load (e.g. images in a page). This might seem contradictory to the above, but a single connection doesn't use all the available bandwidth. There are lots of idle periods during connection establishement and data transfer, waiting for handshake packets on the slow link. So by opening several connections and handling them in parallel, more of the available bandwidth is actually used leading to shorter overall load time.

Having responsive connections is good to speed-up transfer, but even better is not to have to transfer data at all, both for response time and user phone bill. So HTTP cache headers must be handled with great care, which also means that the J2ME application has to implement an HTTP cache.

Cache headers for static resources like images or CSS files are most often handled by the web server itself, by means of Last-Modified and/or ETag headers. You have to be careful with ETags though in server farms, since some web servers include the file inode in it, which essentially makes its value different on every server for the same replicated resource, thus breaking the intented purpose.

Static resources can also benefit from using the Expires header, or better the HTTP/1.1 Cache-Control header that avoids the date parsing and clock synchronization issues associated to Expires. The use of versioned URLs even allows to set an infinite expiration date.

But cache headers for applications are way more tricky. There is often no concept of "last modified", and computing an etag that reflects the state of all the elements that contribute to the response is not always possible or would lead this concern to creep into all application layers.

So I've taken a different approach in the Goojet backend: when receiving the first byte of the response body, a servlet filter checks if the application has set cache headers. If not, the response is buffered so that we can compute a hash code once the application has finished producing it. We could have used MD5 for the hash, but it's a bit costly to compute and we don't need something cryptographically secure. So we use a 64 bit FNV-1 hash that is very fast to compute and has a low collision rate, even for small changes in the data.

The result is that even for highly dynamic responses, we are able to provide cache headers that allow the mobile application to issue conditional requests and download data only when actually needed.

All these techniques combined really make a difference to have a more responsive application and a lower phone bill!

Goodbye Joost, hello Goojet

2008-04-05T11:49:32Z

I have been working at Joost for the last two years as architect of backend systems and tech leader of the backend development team. I started there at a time when the architecture of the system was a blank paperboard. Exciting times, that allowed me to use my creative thinking and build amazing stuff with an amazing team. But also exhausting times, that got me quite burned.

For the last 6 months or so, Joost has been undergoing many changes: organisational changes, stragegy changes (not yet publically visible), and geographical changes by concentrating the previously distributed teams and pushing west towards the US.

So all things considered, it was time for me to move on, and I was helped in that by a startup in Toulouse that I've been knowing since its inception because Anyware was participating to the development team. And since their inception, they wanted to have me on board.

So here it is: yesterday was my last day as a backend systems architect at Joost, and monday will me my first day as the CTO of Goojet.

Goojet is a widget platform and social network targetting mobile phones. Contrarily to other social networks, Goojet focuses on the collaboration between you and your contacts rather than you exposing or broadcasting information to your contacts. The world is made of interactions, not only of yelling at the masses.

My role there will be what I do best and like most: being a think tank, architect stuff, use my synthetic mind and teaching abilities to help the business and dev team understand each other, develop some of the tricky parts and generally give technical guidance and advice to the whole team. I also plan to participate in the nascent or ongoing standardization efforts in the widgets and mobile web domains, but I will first be heads down pushing our first public release out.

The really new technology for me at Goojet are mobile phones: I've been playing with J2ME on my free time for a few weeks, and it is kind of refreshing to work in a very constrained environment. Every line of code counts, no big framework, no high-level abstractions. And the device fragmentation, which requires careful engineering and testing. But it's fun!

So I'm pretty excited by this new job. I know it will be hard and demanding, but well, it seems quiet and easy jobs are not for me.

Cluster computing commoditization

2008-04-02T13:27:10Z

I came across an interesting report on the first Hadoop Summit that happened last week. Hadoop (an open-source implementation of Google's distributed mapreduce infrastructure) is getting a lot of steam, and there are now higher-level open source projects emerging on top of it that I wouldn't even haven't dreamed of a few years ago.

This is a perfect example of commoditization at work, and actually multiple commoditization trends nurturing each other: the cluster physical infrastructure is being commoditized by people like Amazon with EC2 (now with static IP!). Setting up your own cluster which previously required careful planning, good sys admins and a lot of money is now a few mouse clicks away, and Hadoop provides the software infrastructure to easily use this computing power for complex processing of large datasets.

And now that the basic blocks are in place, people are looking at the higher levels: Mahout builds the machine learning tools (classification, clustering, etc) that are so useful when you have to process large sets of user profiles and activity logs to give value to any social network website, HBase to store huge amounts of data in a cluster, and some high-level query languages like Pig or Jaql. What will be the next higher-level that will be addressed?

When I introduced Hadoop at Joost more than 18 months ago to process the user activity logs, it was still very young and a bit shaky. It's nice to see it maturing quickly and getting so much interest.

For sure not every project needs that, but when you're building a social website you have to consider this kind of tool to face the explosive growth that you expect to see happen. It not always happens of course, but if it does you have to be prepared for it!

Novillero, our new (big) pet

2008-03-02T23:35:45Z

My wife Claire has been horse-riding for years, and wanted her own horse for her 40th birthday. That was 3 weeks ago, and the birthday gift arrived almost on time a few days ago. It takes some time to find a good match between the rider and the horse!

So we have a new pet (sort of): Novillero, a 7 years old lusitanian, a species from Portugal (click on the picture for a larger photoset). He is a very responsive horse, which will allow Claire to make a lot of progress, but is also very stable and forgiving.

Of course the horse won't live in our garden! Even if quite large according to french standards, our 2500 m² are way too small for him! He will live in a horse club 10 minutes away from our home, where he will also have a social life with other animals of his kind.

I'm not sure I will ever sit on his back though, since I started learning how to ride only a couple of months ago and this kind of horse requires an experimented driver :-)

A new Ant/Ivy committer!

2008-02-26T14:31:16Z

My dear colleague and friend Nicolas Lalevée has been elected committer on Ant/Ivy, the dependency management tool that allows the use of Maven repositories without the pain of Maven, for his work on the Eclipse plugin for Ivy.

I've been working with Nico for two years, and he's one of the main guys behind the Joost search engine, powered by Lucene, another popular Apache product.

Congrats and welcome in the big Apache tribe!

The hidden gem in the Wii remote

2008-02-08T21:47:08Z

Did you know that the Wii remote contains a 1024x768 pixel infrared camera? With this knowledge an a bunch of IR-emitting LEDs and tape, Johnny Chung Lee, a student at Carnegie Mellon, has done some truly amazing things. Multi-touch screen, Minority Report-like interaction, 3D scenes controlled with you heads movements, etc.

Oh, and of course Mac addicts should try Darwiin-remote to use the Wiimote as the Mac's mouse (or whatever you want). What, you didn't not know that the Wiimote is also a Bluetooth device?

If you don't have a Wii, the remotes are sold separately and are quite cheap. Kudos to Nintendo for having invented such an amazing device!

Skype and the global warming

2008-02-05T12:41:42Z

Skype is constantly eating 10 to 15% of the CPU on my MacBook, and makes it hot even when I'm doing nothing. Is it because I have close to 200 contacts and usually more than 15 open chats?

I have about the same number of contacts on Adium, and it usually stays in the 0.5 to 1.0% CPU range.

There should be an environmental ranking for software, according to the amount of CPU they use while just waiting...

Google opens social graphs

2008-02-03T10:29:52Z

Google is shaking the social networking arena lately. After OpenSocial, they unveiled yesterday the social graph API.

What is it? Basically a specific use of the complete set of hypertext links they build while crawling the web. The links between people (or their website) described using XFN or FOAF can now be searched to discover who is linked to who, and on what site.

Practically, this means that when you register in a new social network, that site can ask Google who your relations are, and tell you who among those are already registered on this new network so that you can more rapidly rebuild your network in the new site. This service can also allow you to know what sites your friends are registered in, which might also be of interest to you.

It does have some serious limits though, as only public relations are indexed since Google obviously don't have your credentials to index your private data. A social graph is a valuable asset and can reveal quite a lot of private information, so many people and websites keep it private.

Maybe there will be a way with OpenID and OAuth to allow Google to index your private graph on behalf of you, but are we ready to let Google know so much about us?

Anyway, it's an interesting step forward that will push the adoption of FOAF and XFN (yes, I have to write my FOAF file!) and ultimately help data portability.

Anyware Technologies bought by Wavecom

2008-02-01T12:27:42Z

Anyware Technologies, the company I've been co-founding more than 7 years ago was bought by Wavecom today. Wavecom is a manufacturer of embedded wireless communication modules that wants to expand its activity higher in the application stack to provide end-to-end solutions.

Anyware's business, which was initially focused on webapps for the IT domain has been growing organically over the years, expanding to a number of domains as opportunities emerged: web applications, Eclipse plugins for specialized IDEs, CMSes, M2M systems to monitor and control remote devices in the field, and consumer applications such as Joost or Goojet.

Although it was sometimes hard to explain customers what Anyware is doing, this is precisely this diverse set of skills and activities that interested Wavecom, because it brings answers to most areas of the full application stack that Wavecom wants to build.

Anyware stays an independent entity wholly owned by Wavecom, continuing the work with its current customers, but will also become the R&D center for Wavecom's end to end solutions.

It is interesting to see hardware players moving higher in the application layers. This is allowed by the increased connectivity and processing power of these devices, and is just the beginning of the world to come where all your devices will have a virtual presence on the internet, allowing them to interact and also allowing you to manage and control them from wherever you are in the world.

Happy New Year!

2008-01-02T11:14:48Z

Happy New Year to you all, known and unknown readers!

This year is not yet 2 days old, and there already has been a big change since last year: it is now forbidden to smoke in bars and restaurants in France. Woohoo! No more painfully enduring bad smell or "sorry the non-smoking area is full but we still have room in the smoker's area".

What compact camera would you recommend?

2007-12-06T14:36:22Z

This is a request to my dear readers. I'm currently looking for a replacement for my Canon Ixus 40 camera. I've never been very happy with its image quality, and lately the lens is doing scary mechanical noise like some missing teeth on gears, and it seems to be loosing sensitivity, pushing the ISO to higher values, thus increasing the image noise.

So I'm looking at the current compact cameras, and nothing really stands out of the crowd. Manufacturers have been pushing resolution to incredible values (up to 12 Mpix) for small sensors, leading to quite noisy pictures when you go past 200 or 400 ISO.

So far I'm leaning towards the Panasonic Lumix TZ3 (reviews here), which has a reasonable 7 Mpix resolution, seems to have decent image quality and an amazing Leica 10x stabilized zoom lens. Definitely not as compact as the Ixus 40, but the additional size for the 10x zoom seems to be worth it.

So, dear readers, do any of you have some advice about a compact camera? What I'm looking for is primarily good image quality even in low light conditions, and a stabilized lens. A zoom better than x3 is a nice plus, but not that important.

Oh, and don't tell me I should get a DSLR. I have a Nikon D40x, and the compact camera is for situations where carrying the DSLR is not convenient.

Android: Google disrupts the mobile world

2007-11-13T09:57:29Z

Google's annoucement of the Open Handset Alliance a few days ago wasn't that exciting: an interesting group of partners and some interesting perspectives, but the meat was missing. It arrived yesterday with the actual release of the Android SDK. And... wow!

This is a complete stack, from the kernel (Linux, of course) up to a very capable web browser (Webkit, also used on the iPhone and the Nokia's) up to the application layers. And the application development is done in Java... or is it?

Google had an incredibly clever idea: Android doesn't run Java, but uses Java bytecodes as an input for a compiler targetting an non-Java virtual machine called Dalvik. Android doesn't claim to be Java compliant (and even less J2ME), and as such doesn't have to pay licence fees to Sun. Moreover, Android ships with a good subset of the J2SE standard library they obtained from Apache Harmony (see also Stefano's post).

So what does all this mean? Google has addressed the mobile market the other way around than Apple: the iPhone is gorgeous hardware with a completely locked system that will eventually allow 3rd party applications in February. Android is a completely open platform, with nice development tools that doesn't yet have any hardware to run on. But we can expect that very quickly, a lot of asian companies which are so good a building hardware but so bad at writing software will embrace Android and provide us with a tsunami of new cool devices, that thousands of Java developers will feed with cool applications.

So with Android, Google is pissing off a lot of people: Apple whose iPhone will stay in a luxury niche, Sun who won't get big bucks from a successful mobile Java-ish platform, and of course Microsoft since we can assume the clunky Windows Mobile will quickly vanish. OpenMoko is also probably stillborn. Now what is Google's intent? Is it just about commoditizing the mobile application market to expand the reach of its advertising platform, particularly with location-based ads? Is it about knowing all about us to increase the targetting and thus the ad selling price?

After OpenSocial, it looks like Google is taking over the platform market. But with open source software, meaning the battle moves to another front, that of our data. This is not less dangerous since data is much more valuable than dumb software. But at least we can hope independent storage services will exist for the privacy-conscious among us.

We want Java 6 on the Mac!

2007-11-09T10:00:32Z

13949712720901ForOSX

Yes, like many others, I want up to date Java support on the Mac! I don't care if it comes from Apple or Sun, but I want it. The Mac is very popular in the opensource Java development community, and these über geeks are very visible (just look at conference pictures) and are purchasing advisors in their work places in "business oriented" departments, i.e. not limited to the creative workers, and also in their families (my parents, sister and parents in law each have several Macs).

So Apple, read this: if you neglect Java developers, you will loose marketshare.

QuickSilver goes open source

2007-11-06T16:10:32Z

There is one utility I couldn't live without on my Mac, it's QuickSilver, that allows me to launch/open/reach all the stuff I routinely use throughout the day with a few keystrokes. An incredible productivity booster that up to now was a closed development by a unique (although brilliant) person.

I just read on TUAW that QuickSilver is now open source! Yay! And even better, it uses the Apache License. I now know that QuickSilver will be there for a long time, for our pleasure and productivity.

Many thanks to "Alcor", it's author. QuickSilver rocks!