Dinesh Harjani's thoughts

Week 7 - The Unexpected Passenger

Mon, 20 May 2013 19:21:40 -0400

(Note that the Pebble watch is not the “Unexpected Passenger” mentioned in this post’s title)

I know I’m late by a day, but I was busy until yesterday finishing the latest touches before wrapping up. It doesn’t feel like it’s my first update since I held off from programming a full 7 Days, mainly because after a day I was completely into it again, and because right now it feels as if I never stopped, and that I completely did the right thing back then.

I have to warn you, though. There won’t be many pictures to showcase this time, mainly because I didn’t feel the need to draw a lot more, and because the project as it is right now is in an inconsistent state: the Engine is kind of disabled for timeframe refurbishments, so I can’t run the new code, and even if I did there’s nothing new to show. But some interesting work was done, and I can’t wait to tell you all about it.

On Monday we begin. My first order of business, as you know, was implementing a serial architecture for operations on the Standard Engine. I decided in the end this would not be a feature of the Zealot Engine design, so the SE had to keep the external methods required by the ZE Objective-C protocol, but run them asynchronously by using the Protocol methods as an Interface to the real methods which are wrapped in dispatch_async blocks. This did not present a problem for the main job of the Engine: refreshing the Track. As it was, we already architected this to work asynchronously with the ViewController receiving a Notification when the job was done. However, the Track filtering system did not work in this way: it was a synchronous method, you called it the moment you wanted the Track filtered and left with your response. This obviously needed to be changed, so I made Track filtering an integral part of how the Standard Engine works: you set how you want the Engine to filter the Track, and it keeps a filtered Track available at all times. When you need to change the filter setting, you do so and you’ll get a Notification to update the Track. Naturally, whenever the Track itself is updated after a new Pool has been processed, the ViewController gets a newly filtered copy of it, true to the filter setting. This is important, because it will eventually allow the user to filter the Track by Supplier, so if you only want to watch Facebook content, you can do so and still refresh the Track. Or Twitter content. Or Tumblr content. Or whatever.

This eventual feature was of curse inspired by Tweetbot and by Google+ for iOS’s management of Circles, only this design will perform admirably quicker, because we already have all the information we need: it’s just running through the Array, writing down the indexes of the Tickets the user wants, making a new Array out of them, and displaying them. No Network requests involved.

With the Track issues solved, after making sure all current tests were still passing (even though I still need to revamp them for timeframes and so on, they were still useful to find out I hadn’t broken anything I wasn’t supposed to), I continued by transforming the Standard Engine so all of its available methods ran asynchronously. Once that was done, some tests wouldn’t go through because in my testing I rely on a “restart” feature I crammed into the Engine which clears the Track and erases all previous information in the Suppliers (of course, this last task is handled by the Commander for the Engine, who knows who the Suppliers are and can deliver the message) so I could perform several small tests within the same “test” function. The trouble was I relied in the restart to be synchronous since the lines of code coming right after can be rendered useless by the restart if it doesn’t happen exactly where I wanted it to, so the easiest solution bar turning the restart into a synchronous method (I did not because at least for debugging purposes I really want to have a “Restart Engine” button in my app) was to insert a sleep(1) function. As it is, my tests do take time, so three or four seconds here and there doesn’t hurt much.

When I got it all running, I proceeded with some refactoring. Turning the Standard Engine into an asynchronous machine added more puff to its implementation, and it made me feel uncomfortable because I felt it was growing closer to the 1K-lines-of-code mark, with still a lot more features to cram in. To solve this, I looked no further than the Track class I’d built a couple of weeks ago: “isn’t adding Tickets, or a Pool, a task a mutable Track would have to handle? And since this mutable Track is built specifically for the Standard Engine, is it wrong if this is the Class responsible for the Gap Detection & Gap Filling features?” My conclusion was a sound “no”, so I spun off the appropriate code into a SEMutableTrack class, and the Engine became a lot cleaner and coherent. The filtering code also fell down to the SEMutableTrack, which is similar to how you can filter an NSMutableArray and get an NSArray with the indexes you want (in fact, that’s exactly what I do). This opened my eyes to a very important issue regarding Objective-C: there are no protected properties. One thing I love to do is use Class extensions to “hide” my own instance variables from the outside, but this means that if I extend the Class, the Subclass can’t access those ivars: there’s simply no clean way. The Internet suggested I keep the ivars I want my Subclasses to see in a different header file used only by the implementation files, but three files per class seems a bit heavy for me, and it just doesn’t explain why Objective-C won’t allow you to share a property with a Subclass. I believe it’s a very common theme in OOP, and it’s also represents very big hole in Apple’s premier programming language, which according to them, is the #1 programming language right now in usage stats.

SEMutableTrack in hand, all tests successful, there was something else I wanted to go after before diving head-first into upgrading the SE to timeframes: I wanted a Pool class. My main idea was the following: studying Apple’s concurrency guides, I understood better how the barrier I was using to control access to the Pool before being processed in the Engine actually worked. Putting a barrier in a concurrent queue means you have to “wait” until everything else in the queue finishes to get your exclusive access, which in real time performance could turn out to be stupid: it’s simply time lost because when other Suppliers might not even be touching the Pool and are instead busy trying to fetch content off the Web we could be adding content to the Pool and freeing threads/memory. My solution to this was creating a Pool class which, similarly to how the SE now does, controls access to its most precious resource (an NSMutableArray) through a serial queue, only this time, calls would be synchronous. Now, synchronous calls can be dangerous if you’re not careful and you call them on your own thread, but since this queue/thread will be invisible to everyone except the Pool instance, there shouldn’t be issues of any kind; it’s a pattern suggested in Apple’s documentation, using serial queues to control access to your resources and get rid of barriers. I did not think doing so would allow me to get rid of the barrier itself, but I found a way of doing it. Getting rid of the barrier is tricky, because to do so the Pool itself needs to be responsible of telling the Supplier whether the Pool is already “finished” (i.e., all Suppliers have added their content to the Pool), and this should be done in a controlled way so no more than one Supplier goes down the “finished” or “let’s process the Pool” code path (in other words, the critical section). I made the Pool good enough to know when all Suppliers have added their content to it in a very simple way: the Engine needs to register the Suppliers that’ll pour in content into the Pool or else the Pool won’t accept any Tickets from them. So, all Suppliers can pour in their content into the Pool, and then ask it whether it’s complete or not. However, this question needs to be performed inside a barrier, or again, multiple Suppliers might get a ‘YES’ answer, and still, I also need to control the critical section is executed only once, because the barrier code is executed right at the end, so more than one Supplier running that code might run into the Pool saying “yes, everyone came in and left me their candy”. It’s still an improvement, because the Pool is filled out much faster, and the code inside the barrier is merely an ‘if’, but it could be done better, so I made the Pool itself answer whether it’s completed or not when a Supplier fills it, and since this is done synchronously, only one Supplier is guaranteed to get ‘YES’ as an answer, so a simple ‘if’ without any barriers is enough to ensure the critical section is properly respected.

Perfect.

All of that took until Tuesday night, and Wednesday turned into a bit of a slowdown.

I can’t necessarily recall when exactly it was, but both Wednesday and Thursday ended up going a bit slow, and not much coding work took place. I believe I wrote down a few lines of code in paper to stretch out a basic wireframe of what I was going to do and how I was going to implement it, and ah!, now I remember, I found out, yet again, that I’d almost missed another component: the two-dimensionality of the problem at hand. I was about to force Suppliers into the same realm the Standard Engine lives in, which is that Tickets should be delivered from [timestamp X] until [timestamp X-Y], but then it hit me: “what if the Engine tried to process the Tickets of a Twitter user who has only published a single Tweet, and that too, back when the service was born in 2006?” Aha! Suppliers needed to be limited too, so I inserted the timeThreshold: Facebook cannot possibly have content predating earlier than January 1st, 2004; Twitter since 2006, Tumblr since 2009, Instagram since 2010, Google+ since 2011, and so on. It might not seem decidedly elegant, but we can’t just scroll backwards through time without a stop. Oh, wait! Can Facebook have content dating back to the 1900s after its new Profile debuted in late 2011? I guess it does… a much older timeThreshold for Facebook it is, then.

Finding out I did not have everything figured out, I prompted myself to look into the future APIs I’d be relying on to form my Suppliers (Facebook, Twitter, App.Net, Tumblr, …) and check if my Supplier API was adequate enough: the issue being I’d been basing my timeframe idea on Facebook’s own API, which as I found out in my Android detour, allows you to set both upper and lower time limits to the content you’re requesting. It turned out this API was mostly correct, only that most services required an ID as an anchor to follow on, not time. So for example if I want the Tweets coming after Tweet Z, I need Tweet Z’s Twitter ID, not its timestamp. If my API worked giving you the key Ticket you were requesting Tickets after, everything seemed OK. Then a small idea hit me: these services are non-linear, I cannot expect them to provide me with Tickets starting from Tweet Z going all the way to the previous 5 minutes. For example, if I wanted to plug in an RSS feed as a Supplier, this would be a complete waste of time, because if I query an RSS feed, the first 5 items will be separated by a span of 3-4 hours at best, and my Engine might start querying the supplier for timespans of no more than 5 minutes: this is a huge waste of bandwidth!!! And the whole architecture would come down to a halt if this were the case. Even worse: if those RSS items are from 2-3 years ago, my Engine would take a good number of requests to get there (I have some kind of regression algorithm in mind to speed things up, don’t worry), and it does not seem like a good design at all.

On Thursday I dribbled a bit more to get my ideas sorted out, and eventually decided building a Cache was something I could definitely avoid, since for the moment I had Facebook and Twitter by my side, and a Cache subsystem might take me a full week to turn into reality.

Friday morning was one of those big, big, big “reality checks”. It dawned on me I really was in trouble: a single Twitter request could give me Tweets from either the last 5 minutes or the last couple of hours/days, in any case, if in the first pass the Engine determines the Pool is not full enough and it can get some more content in, it’ll end up asking the Twitter Supplier for more content: content it received in the first pass, but because there was no Cache in the middle, yet another network request would need to take place, unnecessarily.

Yes, I did. This Cache system is very, very, very necessary. So I got to work. If memory serves me right, principal coding (almost like in films!) starting on Saturday morning. I think I might’ve scribbled something into Xcode on Friday evening/night, but the main ideas started to get laid down on Saturday. The main concept is the following: this Cache is non-existent for the Standard Engine. Whenever the SE requests Tickets in a timeframe to the Supplier, those requests are always fed by the Cache, so any content that’s not on Cache needs to be fetched and added to it. This raises a question: “Tweets and Facebook posts can be deleted at any time, what happens then? And how long does this Cache is supposed to last inside a Supplier?” Tough questions. My choice was to leave whatever is published/posted in the Cache: if it gets caught by my app, it’s not my fault. You decided to broadcast something to the world, and if you take it back, the risk’s already taken, and it’s not my fault if it ends up falling into the screens of my users. If those Tickets are at the top of the Track, a quick refresh might kill them off, (Top Filling algorithm will do them well), but if they’re not, then they’re not. We can figure out much later whether this is a bug, a *lack of features* or whatever, but in all honesty, the Web is as quick as a lightning bolt, and like many things IRL, sometimes you just can’t take back what you just said. As for the second question, the Cache’s sole purpose of existence is saving trips to the slow World Wide Web for my Suppliers, so it’s cleared once the Pool has been completed and its contents are fully processed by the Engine. That’s a change in the API I’ll commit when I’m done with the refurbishments.

The basic way the Cache works is this: you query the Supplier for Tickets from X to X-5, the Supplier will ask the Cache if it covers said timeframe, and for the Supplier to do so, it needs to have content at least as recent as X’s own timestamp, and at least one Ticket older than X-5, and the reasoning goes like this. When refreshing the Ticket, at say time XX, we cannot know whether all the Tickets we get from the Supplier stamped at time XX are all of them or not, because if we’re at time XX, more content can be being added at every microsecond of XX up in the Cloud, so we cannot guarantee that; what we can guarantee though, is that we’re covering a timeframe to a moment in the past, because we can ask for Tickets older than the one’s we’ve been requested, thus ensuring we are supplying exactly what we’ve been asked for. If our last Ticket is stamped at precisely XX-5, it could be that said Ticket is truly the last one at that time, but we need to make sure, and for that, we need at least a Ticket stamped at XX-6 inside our Cache to make sure. This design supports a common theme in the Track: what’s at the top might still belong there or might not. We don’t know for sure, but what’s below it does, because its anchored by what comes both before, and after.

Back to how the Cache works: the Supplier asks its Cache whether it covers said timeframe or not. If it doesn’t, a miss is triggered and the Supplier fetches content starting at the beginning of the timeframe (how much time these Tickets span is not up to us; that’s the whole point!), which is then added to the Cache, and the question is repeated again: “do you cover the timeframe requested by the Engine?”. If the answer is yes, the Cache returns the desired Tickets in an Array, and we deliver them back to the Engine. If it’s not, we fetch more Tickets starting at the oldest Ticket currently sitting in the Cache, and the cycle goes on. Simple. I also decided to limit the Cache’s size, so what happens is every time new Tickets are fetched, if adding them to the Cache will make it surpass its maximum size, the Cache will clear out enough newest Tickets, and then they’ll be added. As to why its newest Tickets, the premise is simple: the timeframe will move downwards in time, so if the Cache doesn’t cover the timeframe, we need to add older Tickets, and chances are the newest ones are off the timeframe, so we don’t need them, and can safely clear them out. If you’re paying attention, you might’ve already pointed out a slight problem in my thought process: I did not at the time, but eventually caught it.

So, with this basic scheme for the Cache working, I set up tests between Saturday late night and Sunday morning: trigger an initial Cache miss, ask the Supplier for Tickets that don’t trigger a miss, then force a miss by ending the timeframe the same second the last Ticket in the Cache is, space out the Tickets inside the Cache and test that a miss is not triggered… and then test the replacement (or eviction, as I called it) policy. This led me to an issue: if the Tickets covering the requested timeframe are more than what the Cache can hold at a single time, the design doesn’t hold up, which is the issue I just mentioned at the end of the previous paragraph. The solution? Add any Tickets belonging to the timeframe that are already in the Cache before getting new ones if the Cache can’t cover the timeframe all at once; after adding them, the timeframe to cover by the Cache is reduced until it fits in the Cache’s maximum size, and we’re done. An eventual “Aggression test” revealed another flaw in my design: “what if the Supplier delivers more Tickets than can fit in the Cache during a single fetch?” Actually, this is something very, very rare to happen IRL, since most APIs limit their delivery to 20-50 items maximum, so a Cache size of 30-60 should be more than safe. In any case, I wrote in a function that allows the Cache to grow in size as much as it needs just in case: for such a simple matter, it was pointless to be inflexible in the code. The Cache does have a maximum size to limit the amount of memory used, but it’s not there to crash the program.

That ends the full adventure for this past week. Unfortunately, from tomorrow onwards I need to work a bit on the Java and Android side, so I’m not sure how much I’ll be able to do about our project until next week. I’ll try to give it a look, not leave it behind, and maybe start nailing down the code for the Engine to work in timeframes, but I’m not really sure I’m in a position to make promises for this week. For sure, I won’t stop thinking about it: there’s a lot of work to do still, because all the tests need to be revamped, but I’m not losing hope. If we finish refurbishing the Engine, the light at the end of the pre-production tunnel will be so close, ending the journey would be like a walk through Champs-Élysées.

(Picture by espinozr@flickr. And no, that wasn’t the picture I was originally looking for, but you’ve got to admit it’s got a pretty impressive feeling to it. So close… yet so far away.)

Week 6 - Rest and No Relaxation

Sun, 12 May 2013 14:48:11 -0400

(Cover art by mqken at DeviantArt)

Beginning to write this is getting harder than I thought, since to keep my 17” MBP comfortable to write (50-55 degrees Celsius) and without over-heating I’m watching last week’s episode of This Week in Tech on my iPad next to me and thus getting the closest thing of an idea to what it is to have a pair of Google Glass on and above your eyes.

So, what was it for this week? I guess I sounded pretty down seven days ago, needing a big break or something. To be honest, by Monday I was already feeling the “need” to get to work, but I told myself that was precisely the reason why I had to stop for a few days. Then a couple more days went on, and I started feeling like maybe stopping was not a good idea, that it’d get a bit hard to get back on the groove, but then I started really working towards the kind of problems I wanted to solve, and here I am with a big picture of what the Zealot Engine is missing.

So yes, no code this week. I read Ender’s Game, and I also put my eyes through multiple WWDC sessions on Grand Central Dispatch plus a few Apple Developer Guides on Objective-C, Concurrency, OOP, etc. What we’re going to do is quickly review again what was the main issue which brought me to a halt last week, and after that we’ll cover the main four features I believe are missing from our Zealot Engine concept.

Back to the “timeframe” issue. Going back to how we’ve designed the Zealot Engine from the start, I’ve been making a mistake right from the moment I started to write down the code. Why? Well, because until now, the Suppliers have been behaving too good: every time we’ve been requesting for Tickets, they all deliver Tickets corresponding to the same timeframe. This means all the Tickets received are “expected” to start and end at a certain timestamp: something which obviously won’t happen with a “real” supplier like say, Facebook. I can give you another example: imagine it’s just a week and a half ago and I suddenly add a Supplier inputting Paul Miller’s Tweets; I’d be mixing content from today with the one coming 365 days ago, and because of how the Engine manages its Pool, all the Tickets in the Track would be utterly destroyed, because the Pool ends a full year ago, and if there were any content in between the Engine assumes (correctly) that the Pool would end closer to the top (newest Tickets) in the Track… you hopefully get the idea. That can’t obviously happen: the Engine is the one responsible for handling this issue; it’s its main job. Suppliers only need to worry about “supplying” Tickets, nothing else (for now, at least).

Now, we’ve decided we need to modify the Engine to work on the basis of “timeframes”. But I have a bigger question: “who’s to say this is the right design? What if I missed something else?” It’s a nice question. I went through as many examples as I could, since that’s the only way I thought could find a new use case I hadn’t accounted for before. Of course, it covers the issue I’ve mentioned before, so that’s solved. New question: “how re-usable is the Engine’s current design?” Suppose I wanted to re-use the Engine to display only someone’s tweets. Is that possible? Is it as simple as making a new Standard Engine instance and displaying its Track in an UITableView? Almost. If you’ve been paying attention you should be able to figure it out. Ready? Well, the answer is a simple “yes”. You only need two things: one, you need a Supplier to give you those tweets (obvious), and two, you need to make your own Commander to install that Supplier into your Standard Engine instance, and boom! So yes, it’s perfectly re-usable. This was always a design goal, but I never asked myself before whether the current design was good enough to meet it: it is. Next. “Is the current pull-to-refresh-based design good enough? What if we want to go live-streaming?” Well, it was never my intention to make the Zealot Engine support streaming; I wanted to solve the problem first and make it that much comfortable later. In case it did get good enough or I consider moving it to the Mac, then I could constantly be polling the web for new Tickets. Still, “what if”? Well in that case, the information would still need to come from the Suppliers, in the sense that the Suppliers would be constantly pushing information into the Engine. So internally, the Engine would need a couple of new hooks to talk to its Suppliers. Externally, we’d probably need to enable or disable this feature someway, so maybe a new method for the API, too. But what I don’t see is any need for API modifications: the current API would still need to exist, so the current design could be perfectly expanded to support live-streaming (we won’t enter into how that could be possibly done), without altering any of the existing code using the Engine. That’s also good news.

So, if you wanted to know whether I’ve found something else the Engine can’t do, the answer is no. I might still be making a mistake, taking another big flaw along, but I think regarding the Standard Engine we’re OK to at least get over the pre-production stage. What comes after that, we’ll see when the time comes. Let’s dive in.

Before we begin talking about which features are missing, I’m going to declare an answer to an un-answered question: “how much exactly is the Standard Engine responsible for?” It’s obvious we’re relying on it to provide us the feed of information, but how much more would we want it to do moving (theoretically) into production? I came with a very simple answer: if we look at a potential app from the point of view of screens, the Engine is responsible for the actions you can perform from the main timeline/feed screen. This involves both acquiring the feed, posting to the feeds, and setting a framework for the rest of the app to work on. Yes, I did say “posting”, but we’ll leave that for later, not pre-production. And about that framework, we can set its foundations right now.

That’s our first order of business, actually. Not the “timeframe”. Before getting into that, we need to revamp how the Engine processes what we ask from it. What if in a production application, the user starts requesting a lot of different operations? For example: retweet, refresh and favourite. Or refresh, retweet, favourite and new post. What happens then? Can we do all at once? Well, we probably could, but there’s something inherent (at least for me) in Social apps about posting something new and getting a confirmation saying your post was successful and at the same time seeing your own post in your feed, just like your friends will. This obviously needs to be controlled someway, and the idea I have in mind is some kind of serial queue processing each issued command to the Engine one at a time. So if you ask the Engine to do twenty different things, it’ll do them, making sure the operations don’t conflict with each other. (We’re going to leave the GCD vs. NSOperationQueue discussion out of this one, or it’ll be an even longer post.) To make things fully come around, we’re going to make the Engine share this main command queue to its Suppliers, which’ll be responsible for their own commands, and which I’m not going to think about right now, but for which we’re going to set the foundations now. You can also look at this command queue (haven’t fully defined it yet; I call it the command queue following the Command Pattern which it closely resembles) as the queue to perform any kind of network operations through the Engine.

Moving on, we finally arrive to the timeframe part. This is the problem we’ve discussed before, and I have some code layouts down on paper to make my mind clearer on how I’m going to face this, but I can give you a simple idea: I’m going to use two additional queues (again, not discussing whether I’m going to use GCD or NSOperationQueues). The first one is the one we had before: a concurrent queue to allow each Supplier run at the same time. The second queue though, will be used to handle concurrent access to the Pool, which is the main contention problem we have while refreshing the Track. Instead of using a barrier as I’ve done until now, it’s much more efficient if I keep a serial queue to access the Pool, and it’s also a lot faster. We’ll discuss more about this as I resume my coding duties on the project.

Only the last two features of the Zealot Engine remain: the first I kind of haven’t exactly figured out completely yet, and that’s limiting the time each Supplier has to return with Tickets. This is because I want to insulate the Engine from any failures going on in the Supplier side; say Twitter fails and due to my own coding mistakes the Supplier never returns back to the Engine. The Engine itself never finishes and thus the app becomes unusable? Not on my watch! The last element you already know of and there’s barely not much thought in it: State Saving. Close the app, open it again: the content in front of you should be the same you had when you left. Not having State Saving puts the Engine through more varied situations than if if State Saving were implemented from the start, but that’s honestly a good thing.

In-order architecture to process commands, in-Engine full “timeframe” support, built-in Supplier failure protection (yes, I’m including protection against network issues too, here), and State Saving. They almost sound like features for a new Intel chip, but they’re not. They’re what separates the Zealot Engine and my own implementation of it, the Standard Engine, from being feature-complete, and thus concluding the pre-production phase of this project. I’ll hopefully get back to full-time coding tomorrow, and if not after-tomorrow (little more design time). The next update in this blog should be a fairly standard one, with pictures of an app running and my cries and triumphs getting some new features (and their tests!) implemented.

Until then, make the most of the week, and if you’re one of the lucky ones attending Google I/O, just don’t miss many beats attending sessions and focus on the incredible people surrounding you.

See you next week!

— Epilogue —

In the time that’s taken me write this post’s first draft, edit it, transfer all the pictures to my Mac, sort through them, edit them, and write the final draft in the Tumblr console, I had a small but very important realisation: the reason why putting a “full stop” to my efforts was a pretty good idea is because I needed to get myself away from my own sense of failure. Getting Gap Technology to completely work turned into a daunting task for me due to that non-existent bug, and if’d kept coding without taking a break after realising the huge flaw my design had, I’d be carrying on me a weight that wasn’t necessary; making mistakes is a big part of why I began doing this in the first place, and every lesson I learn makes me more valuable as a human being and as a professional, not the other way round.

Again, thanks for following me in this little journey. And have a great week!

Week 3.5 to 5 - Gap Technology (Part II)

Sat, 04 May 2013 15:23:50 -0400

This is the week where I’m going to admit to myself, and to you, that this little project has become a lot more than I thought it’d be, because it’s now a part of me. It no longer seems to be just “an exercise” or “a prototype”. It’s my work now; part of what I am. A channel through which I’m trying to give form to something entirely abstract (software) using what others judge to be “my experience”. Things got real this week, specially because I went through a hard time to get here, and it’s tough to admit, because in practice, all the hard work was done two weeks ago, before the “Android break” we came to know as my Facebook Newsfeed (yeah, I know it’s “News Feed”, but anyways) Sample code, and in reality it was too, but I crashed against a barrier that, as stupid as it sounds, didn’t even exist to begin with.

We begin this 10-day odyssey last week’s Friday, just after finishing the Sample code and giving myself a couple of hours to rejoice with my first public code in Github. The first thing I did was get to the table, open my notebook, get up to fill some ink into my fountain pain, and draw the problems I had to solve. As we discussed last week, I’d actually done half of the work (if not more) surrounding Gap Technology because making Gap Detection I covered one Gap Filling scenario: syncing the Pool with the Tickets it had below the Gap (in the Track). Completing Gap Filling would be a matter of making some tests to make sure we could also “extend” the Gap at the end of the Track, make sure the Engine was capable of detecting that we’d really run into the end of the Track (Suppliers did not have new Tickets that were older than the ones sitting at the bottom of the Track), plus a couple other cases. Before that though, we needed to solve a big UI problem, which is that so far the only communications with the Engine have been directly between the ViewController and the Engine itself. Filling in a Gap for example though, is very different, because the TableViewCell representing a Gap is not owned by the ViewController, but by the Engine through its Suppliers. How do we get the message from the Cell to the ViewController? I could’ve done a lot of schemes involving delegates or causing all of the Ticket’s interactions to channel towards a method in the ViewController (the didSelectRowAtIndexPath: selector), but in the end I opted for a very Objective-C like solution: notifications.

By “notifications” I don’t mean the ones you see on your Droid, iPhone, iPad or Mac, I mean the internal notifications used by Objects and libraries from the Cocoa Touch framework. Another similar solution would’ve been using KVO or “Key-Value-Observing”, which is very similar in practical terms, but less straightforward. What you do in KVO is tell Cocoa you’d like a selector (method for those who haven’t done Obj-C) to be called when a property (member for those coming from Java) changes. I could’ve used a special property just for triggering these kind of messages, but notifications seemed like a much cleaner and leaner solution. I also wanted the ViewController to control the action of what’s done and what isn’t at all times so, even though the UITableViewCell animates itself to show you’ve “activated” a Gap, in truth it sends a message to whomever wants to listen to it, and the idea is that the ViewController will receive that message notification and then ask the Engine to “fill in this Gap”. That’s the design I liked.

After going through all of that in paper, I spent Friday making that work, and since we were going to play a lot with time-based algorithms, I went ahead and made custom UITableViewCells for the ProgrammableSupply, showing an avatar, the user’s name, the message and the Ticket’s timestamp. Once that was working, I started to re-factor the code I had to Bottom-Fill a Gap so it could be re-used depending on an index. I re-ran all tests to make sure nothing was broken thus far.

Saturday I started by testing. I made the code I had thus far “extend” the Gap we make at the end of the Track: that went without a hitch. Then I made a Standard Gap appear at the top of the Track, and filled it in normally, without triggering any new code, producing yet another Gap at the end of the new Pool. My idea was to test whether the Engine would be able to detect that the Tickets provided by the Suppliers did not cover the Gap we were filling in and to ensure a new Gap, covering less time, would be added at the right spot. This required some changes, since until then I’d been adding Gaps directly to the Track, when in truth the correct way to do it is in the Pool. Why? Because the Tickets in the Pool are the ones guaranteed not to change, whereas the ones in the Track are at the mercy of whether they collide or not in time with the Pool (the “Filling” algorithms we’ll talk about now). So I made that change, had all of the tests run, and all of it was working by Saturday evening. After coming back home from a very long walk and a nice dinner with a friend, I did not go to sleep: I came and I sat in front of the screen. If Gap Filling can be split into two distinct actions, which is taking care of the Tickets above the Pool (Top Filling) and those below it (Bottom Filling), the only thing I had left to do was the former one: so my idea was to take the test in which I’d extended a normal Gap, and fill it with a bunch of recycled tickets to exercise the Top Filling code path. After two hours, I thought I got the code right, ran the test and saw it fail with strange numbers popping up in the Console Output. I told myself it was past 2am and I shouldn’t expect to be in the best shape to be writing code, and decided it was probably something easy I’d be able to fix on Sunday. (Spoiler alert! That was completely true, but it’s not what happened).

Throughout Friday and Saturday I was taking the whole thing slowly: I knew it was only a matter of fixing some small chunks of code and making those tests to get it all done. I just needed to think clearly, go slowly, and get things done. So on Sunday I started looking at that “Top Filling” code, figured I had some mistakes here and there with the indexes, changed them, saw how the tests worked suddenly after shutting everything off except what I was trying to fix, then turned the other tests on, and saw how they failed yet again. More changes, this time reverting back to what I’d done before, and then everything worked, except Top Filling. So I turned all the other tests off except the first two: the first one produced a normal Gap and “extended” it, and the second (Top) filled it. The second one still failed, so I figured my problem was somewhere here. Again, after dinner I decided to give it another shot, and began looking closely at the Simulator, and that’s when things started to go really wrong: it seemed like the algorithm wasn’t working well on the second test not because it wasn’t right, but because the pointer was getting assigned to the wrong Ticket even though its index was right. I went to bed on an absolute low point with a couple of horrible words stinging my mind: memory corruption.

Somewhere, somehow, my first test was doing something it shouldn’t have been doing because the second one did not correctly dereference an index from the Track: it used a wrong Ticket, and thus the algorithm wasn’t working. There were no memory access errors of any kind; no exceptions, and like all memory problems, if I started turning on/off other tests the results started getting completely wild. Suddenly my threads stopped in a Block because they’d lost ALL of their pointers, which led me to believe that maybe because I was passing in blocks as arguments many times and blocks didn’t live on the heap but in the stack, somehow they were being deallocated before time. Yes, it started to get very, very “Wild Wild West” out here. I mean, I cannot explain it to you in words, but I really started to throw a lot of hours at this: like 6-7 or perhaps even 8 in front of the screen, plus more when I was not in front of the screen trying to tackle the problem. I began to realise Xcode was showing a lot of 0x00000000 or nil pointers in the Track after “extending” the Gap, which confirmed to me that something was going awfully wrong because all of those were strong pointers to my Tickets, and I believed inserting new Tickets was working all right, so no wonder my algorithm was failing because it must be accessing a corrupted section of memory or something. On Tuesday I ran Instruments and activated every single memory detection checkbox there is in Xcode, and nothing came clean. Nothing. To run Instruments well, I even made the same thing happen in the Simulator to get a reading on memory leaks, but there were no leaks except a few KiloBytes which I thought would most likely be auto-released NSStrings: the Tickets should leak more memory. Adding insult to injury, Xcode started to crash, too, just where you would guess: executing the Top Filling algorithm, at the precise line where I can check through the instance variable’s memory addresses whether the Ticket was assigned correctly or not.

Despair seemed to be within my grasp.

Wednesday I tried to focus on thinking differently: revising documentation about ARC, memory management, about GCD blocks being released prematurely, about array assignments being set incorrectly: the latter three did not have any results on Google. It briefly crossed my mind that maybe, since Top Filling was only triggered when the Suppliers were recycling Tickets (adding Tickets from the previous Pool into the new one) and that disabling recycling made everything work, that perhaps the error was somewhere lying in there. I gave it a look, and went a bit crazy doing hard-copies of the Tickets: I thought maybe they were being de-alloc’ed because ARC wasn’t holding on to them after the Suppliers stopped keeping them for recycling due to some ridiculous pointer referencing cycle (thus, why so many pointers were zero’ed out). That didn’t explain why the whole system worked in all of my previous tests, but who knew? That didn’t work, so I looked at when and how the Track inside the Standard Engine was getting filled with zeros: it was after the Pool was being inserted in place. Previously, I’d always been adding the Pool right at the end, but this was no longer possible since the Engine supported filling in Gaps, so it looked like this code was somehow responsible for at least some of the errors, even though I made it following Apple’s own documentation. I changed it a couple of times, and in the last one I even built a whole new NSMutableArray composed of the Pool and the previous Track to form the new one, and voilá: Xcode showed the right results!

Wait. Wait. Was Xcode playing with me all of this time?

It was. I realised that when after re-running the algorithm it turned the exact same results as before: the algorithm had always been working and there’d never been any memory corruptions or leaks. My blocks were being dropped inside GCD because by mixing code I was telling the Engine to fill in at a non-Gap position: something it was not designed to do. (I admit to not following why exactly that was happening, but it wasn’t the Engine had to do: that’s why we came up with Gaps to begin with.) And the algorithm was not working because… guess why? The recycling code had a bug! I added new stuff to the Programmable Supply so it could either recycle the newest or the oldest Tickets from the previous Pool, and since from one Pool to the other you don’t know which Tickets are going to be recycled and which aren’t, the Tickets in the recycling Pool inside the Suppliers were out of order! That’s why the Test failed depending on what was done before: because the Suppliers weren’t recycling well. I added the same ordering code the Engine uses (3 lines), and… care to believe it? It worked!!!!!!

What a great way to end Wednesday night. It was all solved: Top Filling worked, Bottom Filling worked. Now all that remained was putting up some tests for filling in a Standard Gap and we were done!! The single most-complicated feature of my Standard Engine had been solved!!!

That’s where true happiness ends, though. I finished everything by Friday night. I took it all slowly, made sure it all worked, even made a couple of extra tests to handle a few difficult situations, but on Wednesday night, after walking my dog out just before heading to bed, I realised my Zealot Engine’s concept and API was heavily, heavily flawed.

This is the same thing that did not allow me to solve Gap Detection quickly: thinking in only one or two dimensions. Consider this: the Engine is asking its Suppliers for Tickets. They deliver, and the Engine forms gaps if the timings don’t match. However, I’ve been relying on how all of this works in my perfect little world, not in the real world of different services and many, many users. The thing is: if I ask Facebook and Twitter to tell me what my friends might’ve posted recently, Facebook might have 180 items from the last 20 hours, whereas Twitter might have 180 items from the last 20 minutes. This is not right, because the Engine will form a timeframe lasting twenty hours, when in reality many Tweets are clearly missing.

This is a very big flaw: I could’ve decided the Engine is not responsible for this and that the Suppliers should control the timeframes every Pool represents, but that’s just nuts and stupid: it means making a lot of code that won’t work when simply placing that code in the Engine would make the system work perfectly. This is an issue that leads to all sorts of problems, and the truth is, if you are building a Social Media app or an engine that only works with one source, you don’t have to deal with this, because the timings are be enforced by the content you receive, not by your business logic. I messed this up the day I began this project: I messed this up the very same day on Week 2 when I drew all the interactions and didn’t stop to think about this. I’ve been putting a wrong foot ahead of another ever since Day 1 of Week 2.

It’s a big blow, because it’s time I’ve lost in this due to my own fault. And after Gap Technology, the remaining features are actually not that hard: State-Saving and protecting/preparing the Engine for Internet connectivity and unresponsive Suppliers. Had I taken my time, I wouldn’t have to re-work the whole API and re-write each and every one of my tests so far. My goal of finishing the Standard Engine in this month of May now seems farther away, since I honestly don’t know how long I’m going to take re-designing the API to work on timeframes. You could come to me and say: “Duuuuude, just refactor the hell out of it and be done at the end of a day!” Yes, that’d be the spirit. But I messed up once and don’t want to mess up again. I’m really going to take it slow now. I need a couple of days doing nothing, and by “nothing” I mean no code and no fountain pen for schematics. When I feel OK I’ll start looking into a couple of apps, see if there’s something else I might be missing, come back, get to the drawing board, fix the Zealot Engine, and then proceed refurbishing this ship.

So that’s what’s ahead. I don’t know how much work you should expect from me this week, because I don’t have expectations from myself. I need a little bit of perspective, just a little bit, because this is just too close to my chest right now. I’ll get back to work as soon as I can; I’m not abandoning this, I promise. But I am sorry for letting myself and you down if you’ve been following my progress thus far.

Enjoy your weekend. Because, I’ll be back.

Android Detour: Facebook Newsfeed Sample

Thu, 25 Apr 2013 12:57:00 -0400

Right, so last Sunday I mentioned I’d be spending a few days doing Android code instead of following up with our project. I look forward to actually get going as soon as possible, so before that, I thought it’d be a good idea to write a bit about my first ever piece of open source code on the web.

The issue prompting me to write this code is that many companies like to see some code before they trust you enough to sign you or even for an interview: they want to know how good you are, and if you don’t have apps out there and you call yourself an “Android & iOS Developer with 3-year experience”, well, you’ve got to back it up with something. I’m not going to lie: I didn’t want to lose focus on what we were doing, but I took the chance to do something I hadn’t done since a very long time, which was working with Facebook’s API.

The idea for building a Newsfeed sample code on Android was pretty straightforward, since it was the closest thing I had to what I’m currently working on and it’s also the best opportunity I have to get some prior experience with Facebook’s Graph API before the day I (hopefully) have to call it down on iOS. I was a bit scared, though: I thought I’d take a lot longer. I had no idea of how long it’d take: in my best guesses I’d be writing this text on Friday morning, not Thursday evening. Starting on Monday, I took it easy: spent some time on the web, started revising some Android code, some Android books, and started by opening up IntelliJ-Idea and wiping out some code for an Android app that worked both on the old Android and the new one. Something I’ve kind of done before, of course. Then I realised the first thing I had to do was dive into the Facebook Documentation, so I did.

In the past, things like the Twitter and Facebook Documentations seemed incredibly daunting to me; I mean, they’ve got everything you might want to know, but they don’t set you on a path, they just throw you into the Ocean and expect you to find the way towards what you’re really looking for. Which is fine, if you exactly what you’re looking for. Fortunately there are some tutorials and sample code here and there, but they’re a bit sparsed throughout their sites. As always, “the code doesn’t lie”, so going through Facebook’s sample code for Android made me realise these folks have actually done a ton of work: not only for users of the Facebook API, but for any Android developer in general who’s willing to take the little bit of time it takes to grab a part of the SDK and adapt it for their uses.

Tuesday morning, realising this was going to be a Github project, I started anew in a folder containing the Facebook SDK and the project itself. I started by what you’d expect: logging in and showing my Profile picture. Once I got it running, I shifted the code a bit to specifically use an API Request to do that instead of relying on the LoginButton, and when that was done, it was time for the big show: downloading the Newsfeed.

I looked through the Documentation: no specific information about how to get my Newsfeed. No help on the web also. Looking into the SDK’s methods I found I could make specific Graph API Requests, and having played with the Graph Explorer, I already knew exactly which address and with which parameters I had to work with. A little step here, though: I was writing everything into the address without taking care of URL encoding. Finally some code snippet I found around the WWW led me to introducing the parameters as a Bundle, and voila! Facebook started giving me my Newsfeed in JSON format.

But the app was crashing. The one thing I knew would be a bit tedious to code was JSON, because as much as I’d rather work with JSON than with XML, JSON still requires a ton of boring old code to work with. Facebook apparently solved this: they automatically generate a Java Object strongly typed against JSON Object’s parameters through a Java Interface. In other words: all you’ve got to do is give a magic Object an Interface through which you want to access a JSON Object’s properties, and from thereon, you can treat all the Java Objects as if though they implement that Interface. The Documentation mixes it up a bit when referring to “Classes” and “Interfaces” but ultimately I got it right, after finding out they made a mistake: the “id” property requires a getID() method, not a getId(). What do I mean? Well, to “generate” that code, Facebook relies on the names of the getters you specify in that Interface: it’s all what you’d expect it to be, really. The “name” property is “getName()” method, “description” is “getDescription()”, and so on. The only exception is the aforementioned “id”.

Once that was solved, I learned that I could also “chain” these so called Java proxies, in case a JSON Object had another Object as a property, which is nothing sort of wonderful. Wednesday was the day I brought all of this together to form a real working sample code, complete with a fancy looking row, an Adapter, a Wrapper or “ViewHolder”, and an AsyncTask to query Facebook for the Newsfeed. Because I never have enough, and because I’ve always wanted to do it, I used an as-of-3-months new member of the Google Family’s Pull-to-Refresh implementation and made it work to refresh. I used some of the experience from the project we’ve been working on to get it all working just as we expect: new items at the top, and so on. There was no need for any fancy Gap Detection code, since all I had to do was to ask Facebook if there were any new Newsfeed Items since the last one we received (if only I were so easy on myself…), and no issues. We order the items, keep them always from oldest to newest, and the Adapter takes care of inverting how we see the items to form an ideal experience.

That was pretty much the full story. After that I added an option to Logout, since the Facebook Documentation begins by Login and Logout, and spent the rest of the time documenting and commenting all of the code. You can get it right here. And as always, I welcome any suggestions, comments or additions you may have :)

Week 3 - Gap Technology (Part 1)

Sun, 21 Apr 2013 18:18:09 -0400

Sunday. I’m back, and it’s honestly been a long week for me. I had a lot of fun doing this, and I seriously thought I would either not hit my goals or not dodo as much work as I wanted to. For the first time in the last 21 days, I failed my own expectations, but given how complicated everything came out to be, I’m really really glad about how things ended.

Are you ready? This might get long again. Just to be sure, I won’t be writing this in the Tumblr site itself but using iA Writer for the Mac :)

Let’s begin. On Monday, after finishing Part Two, I needed a rest. That meant watching a little bit of TV, specifically Game of Thrones (thrilling episode, although nothing important happened except a baby boy was born) to rejuvenate a bit, and then going back to coding right after dinner. Now, I don’t know if I’ve mentioned this before, but the requirements of the Zealot Engine are marked by what I consider the best social apps in the business, most prominently Tweetbot for iPhone. If this ever turns into an app I won’t make it into a Tweetbot clone, but I want it to stand pretty close in terms of quality and features. One such feature I like from both Tweetbot and other great iOS apps such as Google’s own Google+ client is the ability to “switch timelines”, and by this I mean (if we consider the Timeline to be the list of items you scroll through on the screen) that you can choose to see as much information as you can get (your full stream on both Twitter and Google+ respectively) or restrict it down to only some specific sources (lists and circles respectively again). This was a feature I did not remember about during the work period in Week 2, but it was clear to me over the following weekend, so it became first order of business in Week 3, and so I pursued it.

The code itself was pretty simple: you can still ask the Engine for its full list of tickets, or you can ask for the list of Tickets filtered through a source. This is where the ZE implementation starts to get tricky, because no matter how “abstract” everything is coded, it’s inevitable that at some point in development you’ll have to write some code that knows about class specifics and/or is tightly coupled. If you don’t, nothing will ever work. In this case, I decided to go for enums, so the Supplier protocol itself defines an enum with possible sources, [the enum] itself defined as an NSInteger with different values. What this means is that if somebody else wanted to use the Zealot Engine, they’d either have to add their own enum values or change the ones I’m using. If there is ever more than one “Zealot Engine” and it’s not made by me, I think it’d be OK if they used their own sources with different values. It’s a Java-like approach, but for the scope of the work I’m doing there shouldn’t be any problems. If you’re asking why this has arisen now, it’s because when performing filtering over the Engine’s current list of Tickets I had to filter using some kind of specific information, and the other idea I had was using specific classes for each Ticket. That seemed a bit messy for me, so clean and classic “const” strategy (enums in this case) it was. The code itself was pretty straightforward: when asked to deliver a filtered list I used an NSIndexSet and used the “indexesOfObjectsPassingTest” method to perform the magic. Pretty simple to write, and pretty simple to test. I did all of this on Monday night.

(Image found here.)

On to Tuesday, but before we get to that, I’m going to add a new name to our current mix of Tickets, Users, Suppliers, Engine(s), Pools and Commander(s): the Track. The concept of the Track became pretty clear to me at the beginning of this week, but I did not implement it until yesterday night past midnight (which is technically today only 17 hours ago), and what it is, is an abstracted look into the Tickets currently being held by the Engine, but much closer to the UI than to the Model. The Track itself is a concept: the “list” of tickets being held by the Engine, but I defined it as a class which is the one Engines communicate with, not NSArrays. This means nobody knows outside of the Track that what it really does is invert how you read the Tickets off of it, so that functionality is now performed by the Track itself, not by the ViewController making use of our Engine. This is actually something I should’ve done since the beginning, but I’m happy to have figured it out pretty quickly and to have performed this refactoring during this week.

As to why it’s called a “Track”, it’s because if you can picture the sides of an iPhone’s screen as the rails through which the Tickets keep on coming, the Tickets themselves look like the wood a railroad track is made out of in a classic “railroad shot”. To review the full Architecture a little bit, the Engine keeps the Track, builds a new Pool of Tickets, and when the Tickets forming the Pool are ready, they become a part of the Track. Onto the real meat of this week now.

Like I’ve said, many of the Zealot Engine’s features are demanded by what I think any user would expect from a well-crafted application. One of the most lauded features of Tweetbot in any of its variants (iPhone, iPad and/or Mac), is the way it keeps track of your Twitter Timeline throughout your use of it. Specifically, it has a way of representing all of the tweets in your Timeline in a very simple way to understand: gaps. Tweetbot adds gaps to your Timeline to let you know there are more tweets between the one above and the one below, and that you can see them if you so decide (by tapping on the gap), and there’s also a gap at the end of your Timeline so you can keep scrolling down through your Timeline infinitely. Out of all of the functionalities the Standard Engine must have in the list, because this is specific to my implementation of my own Zealot Engine family of protocols, this has to be at the top in the complexity list. My thoughts on this were simple: filtering and Gap Technology. Surely they can be done this week?

Back to Tuesday. I already knew I wanted to tackle the “Gap Technology” feature head-on, so my first homework was implementing some way of detecting when two Tickets are the same. By “the same” I don’t mean that they’re the same Object, but that they represent the same information, which is likely to happen when updating the Track. To make it happen in a very straightforward way, I had to take another step forwards: the StandardTicket. Designed as the simplest unit of concrete information flowing through a Standard Engine, the StandardTicket properly implements the isEquals() method for any possible subclasses to make use of, plus a couple of improved setter methods here and there. Making the tests for this new functionality took me some time, and I lost Tuesday to this apparently tiny little thing. Gladly, everything was working by Tuesday night.

Wednesday. My head is beginning to hurt only by thinking about everything that started here. So, Wednesday there was no code. Actually Tuesday and Wednesday were both days in which I barely had any time to invest here, and I actually thought it’d be a pretty horrible week; it wasn’t in that sense. So, what I did on the while I had on Wednesday was pretty much sketch out everything related to what I defined “Gap Technology”. This so-called “technology” was broken into two parts: Gap Detection and Gap Filling. The first one is merely responsible for detecting when a Gap needs to be added to the Track, and the second one transforms a Gap into new Tickets which magically appear in-place in the Track. You must be thinking: “but that’s inefficient in an NSArray!” Hang on, here. The first devil of all software is premature optimisation, and though I did not entirely follow my own rule as you’ll find out soon, this is one of the reasons I “spun” the Track out of the Engine; in this way, I can completely hide how the Engine manages its Track, and if I consider it’s better transforming it into a doubly-linked list, UI performance won’t suffer because the Track the ViewController will be using can always represent the Track as an NSArray: 2/2.

Let’s focus on Gap Detection first: what is a Gap? A Gap represents a sequence in time the Engine has no knowledge about the existence of Tickets. This means the Engine can make the mistake of adding a Gap in time where no Tickets were produced, and it can also mean a Gap can represent more Tickets than the Engine can grab in a single Pool, so filling in a Gap may spawn another Gap, the second one representing a shorter sequence of time than the first. Next, who’s responsible for the Gaps? The Engine. A Gap is not a StandardTicket: it abides by the rules other Tickets do, but it is not a StandardTicket per se; it is entirely managed by the Engine, and thus shall be excluded from most tests. This means the Engine is a Supplier? Technically it is, but I don’t want to burden the Engine by making it compliant with the Supplier protocol and implementing its methods. I still want the outside world (the ViewController) to know as little as possible about Gaps, so I still want the Gap Ticket’s Supplier to handle the UITableViewCell and so on. The solution I opted for was composing the Standard Engine with a its own custom Gap Supplier, which keeps things elegant and at the same time allows me to keep the Engine and its Gap Supplier tight, which is exactly how they should be.

More questions: when are Gaps created? There are two situations here. The first (and simplest) one is the Gap at the end of the Track, which basically stands for all the Tickets the Engine can’t account for in the history of the Universe. The second, and the most common situation, is when we form a new Pool of tickets, and the Pool and the latest Tickets in the Track don’t intersect in time. This means the oldest Ticket in the Pool is still newer than the newest Ticket in the Track: therefore, there may be Tickets in between, and so a Gap is formed. How many types of Gaps are there? Two. The “Infinite Gap”, which is basically the Gap at the end of the Track and which will rarely disappear, and the normal Gap, which happens to cover a specific period in time. Broken down, it doesn’t seem like all of this “Gap Technology” is any hard, right?

Pretty much all of this was written down on Wednesday, but some details were left to Thursday morning, which kickstarted my “full ahead” mode on this problem. I started first with the Infinite Gap, which was pretty easy: after processing a Pool, if the Track is empty, an Infinite Gap goes before the Pool, and we’re done. Then came the other type of Gap, the normal one. Coming into this problem, I thought Gap Detection would be incredibly easy and that I’d be able to leave how to properly detect when Tickets intersect in time with one another for the Filling part; after all, all I had to do was either add a Standard Gap or add the Pool’s Tickets on top and we were done, right?

I started having problems writing down the code that followed, and that should’ve been a signal to me saying: “This is not as easy as you thought it was. No matter how much ink you’ve spilled into this problem, you still need to spill some more.” Many minutes staring from my screen to my notes followed, again and again, until I finally managed to start making breakthroughs by writing code. I thought the hardest part would be making the Gap appear: I was a fool, that was the easiest part. All I had to do is simply check what I said before: is the oldest ticket in the Pool newer than the newest in the Track?” If the answer is true, all I had to do is add the Gap and move along just like we used to do before. Easy peasy. What if the newest tickets in the Track were newer than the oldest in the Pool? No Gap was required, we just needed to add the new Tickets. Good luck with that, Sherlock.

I attempted to write the code. It turned out very big. I never tested it, but it seemed far too big, and going against “optimisation is the root of all devils…”, I rewrote it again, made it shorter and prettier. I ran it: disaster. My Gap Detection was literally eating through the whole Track and destroying a lot more Tickets than it should’ve. To make this whole thing work, I added a recycling functionality to our old friend the ProgrammableSupply, so the second and subsequent Pools would be filled with the last tickets of the previous one, and thus the no-Gap branch in the Gap Detection method of the Engine would be triggered. This worked, although it gave me a couple of scares, but the code I was writing was clearly not doing what I wanted it do do. My theory was pretty simple: all of the Tickets in the Track that are newer than the one at the bottom of the Pool should be discarded, because the Pool obviously carries more recent information and thus has priority over the Tickets in the same timeframe in the Track. After that, we need to find a matching Ticket in the Track, and when we do, we can stop and finally join the Pool in. No deuce. My while loop discarded a lot more Tickets than it should. Then a thought sprung to mind: a lot of Tickets were being created down to the very same second and even micro-second, which certainly that doesn’t happen so often in real life. I added another functionality to the ProgrammableSupply, a delay per ticket. I set it to zero in the tests I had for the features we have already finished, but by default every Ticket took a second to create. Since the Engine is using GCD and spawning a thread per Supplier, there are still many Tickets being created in the same second (3 at a time, because I’m using three ProgrammableSupplies), which is not exactly what I wanted, but it’s close enough to a real-life situation, which I’m keen on following with all tests. No luck: the while loop was still eating a lot more than it should. I shifted from the compare: function to timeIntervalSinceDate:, but things weren’t improving. I finally set up a breakpoint plus NSLogs and found out it was discarding Tickets in the same second as the last one in the Pool, which was clearly not my intention. I modified the code so only Tickets newer (in seconds) were discarded. Still no match: If the ProgrammableSupplies recycled a total of 27 tickets, the Engine could easily eat up 40 or more. Why? I then improved the code that came after that first while loop, the one looking for a “match” to put both the Pool and the Track in sync. No luck. The Engine performed less aggressively, but this had a downside: there were repeated Tickets. Not what I had in mind. Heads down, I called it a day on Thursday Night.

Friday morning. Of course the Engine was being overly aggressive!! The ProgrammableSupplies are each recycling a different number of Tickets, that means they’re out of sync with what the Engine has in its Track, and thus we’re eating more Tickets than we’re recycling. It seemed like that was the key, and I was happy to get to work on that idea minutes after waking up, but I decided to go another route: tests first. I wrote the Infinite Gap Test, the Standard Gap Test, and then I went for the Gapless Pool Test (aka “the problem currently at hand”). I made a different Unit Test for the Gap Technology, since these tests did not run the Engine so many times and because they were much slower, using the Ticket Delay feature in the ProgrammableSupplies. First two tests were clear. I made the third one forcing all Suppliers to recycle the exact same amount of Tickets. I ran the Test once: we were overly aggressive once again.

I couldn’t believe what the Console Output was saying. I ran the app in the Simulator and saw it for myself: one Supplier’s Tickets were always being killed, like ten of them. Something was still very wrong. I ran the Test again, got up from my chair, walked around the house… “Tests Failed” said Xcode. The Engine was now not being aggressive enough, and my Test was detecting that there were repeated Tickets left in the Track. Another walk around the house… and an idea erupted: “You’re an idiot!”. I’d been failing to look at the process in two dimensions: just as the intersection of the Pool and the Track begins at the Pool’s last Ticket, it also ends there. Meaning if while clearing out the Track we find a Ticket that’s older than the last one in the Pool, we’re in sync, because this is information the Pool simply doesn’t have. A very big mistake on my part: the problem was not one-dimensional, but two-dimensional. Throughout all of this time I knew that testing with real Suppliers would’ve not made this design failure so prominent, which irritated me, but once I realised it was a true error on my part to properly assess the problem, I breathed while correcting the “matching” loop thinking it was a mistake I would not make again, and that this was exactly the reason why I started working on a project like this to begin with. I updated the code, I ran it: “Tests Succeeded” it said. I ran it again, not believing my eyes: it failed. I ran it in the Simulator: it worked, no visible gaps (sorry for the pun) in the Ticket IDs. I ran the tests: success again. But something was not fully working, so this being a test, I switched the numbers, and added a second subtest to the GaplessPool: the Aggression Test. In the first subtest the Engine must clear exactly the number of tickets recycled by all the Suppliers, because they all recycle the same number, and the Aggression Test makes sure the Engine behaves aggressively when the Suppliers recycle less, meaning some information was lost, and so the Engine must clear more out of the Track because the Pool’s information prevails. Tests started to fail again because the Engine was missing out a single Ticket. I looked at the debugger, and I reached a painful conclusion: the Tickets were not sorted equally in the Pool and in the Track. This caused the “sync” to happen when a duplicate Ticket was left out in the Track, thus failing the Test for two reasons: one, a recycled Ticket failed to be cleared, and two, a that Ticket obviously ended up being duplicated. To add insult to injury, the Aggression Test was not matching the number of cleared Tickets I expected, which it should’ve, because the problem was already localised and it shouldn’t be affecting this test. I looked into StackOverflow, and changed my sorting function. Instead of using a block and compare: I used an NSSortDescriptor. Tests Succeeded. I tried again. Tests Succeeded. I tried again. Tests Succeeded. Again. Tests Succeeded.

Sorting with Blocks was giving me wrong results? A bug in Apple’s code? That seemed strange enough, until the Tests Failed again for the same reason as before.

What was exactly the problem here? Too many Tickets were being created at exactly the same time, perhaps even down to the microsecond in my Mac (can’t perform Unit Tests on hardware, Xcode doesn’t allow it). Would this happen with real Suppliers? Most likely no. The tests were failing because a single Ticket was out of order, and simple hacks could make this nightmare end. I could make the last Ticket in the Pool I check against move to a newer Ticket when a match fails after being in sync to prevent the sorting error, or I could check after the Pool and the Track are in sync that the remaining Tickets in the Track that are in the same second as the last Ticket in the Pool are not in the Pool (sorry for the mix of words, but it really is like this). I didn’t like either. The first one was a mess, and the second one turned my whole algorithm from O(n) to O(n^2). Something faster than O(n) is clearly impossible, and going to O(n^2) would waste my (possible) users’ battery life because my Zealot Engine is flawed. I didn’t like the feeling of building my Engine on top of a flawed design. Did the guys at HootSuite have a really hard time with this, too? How does Twitter handle all of this? Clearly, the have thousands, if not hundreds of thousands, if not millions of tweets being created and dispatched in the very same second. Who do they handle this? Ah, because every Tweet has an unique ID. How about sorting based on a custom ID?

Let’s see. We want everything ordered by time first, so this means our custom ID will start with each Ticket’s timestamp, and since this is best represented as the number of seconds since 1970 (aka “Standard UNIX time”), we’ll give that 32 bits of space. Then, we want to order everything by source, which for us is our enum. We need some headroom here, so we’ll give ourselves 8 bits, which gives us 256 possible values and brings us to a total of 40 bits of size for our Custom ID. In case two Tickets were produced at exactly the same time by the exact same Supplier (knowing Twitter it’s not that difficult), we need to break things up by user now. Considering Facebook has broken the 1 billion users, we need enough bits for that. 32 bits equals to roughly 4 billion possible values, and now we’re up to a whooping 72 bits. If by any chance a user pushes two different Tweets in the same second using two different clients, we need a Ticket ID to break the tie, which puts us in 128-bit territory for our Custom ID value. 128 bits, or roughly 125 KiloBytes of memory lost per thousand Tickets only in our Custom ID. It doesn’t seem like much, but I don’t want to push 128-bit numbers comparisons into 32-bit register ARM CPUs, so I’m taking that off the table. What if instead of a number we represent our Custom ID as a String? I’d take up a lot more memory.

Admitting defeat, I called it quits on Friday night, thinking the only solution I had was the O(n^2) hack, which compromised efficiency and speed, but seemed like the cleanest solution taking into account that in most real-life scenarios the Engine could process the Pool in the ideal O(n) time.

Saturday morning. I’d dedicated two full days to the Gap Technology problem, three if we count Wednesday, and hadn’t managed to even finish what seemed like the easiest part. I said I’d use Saturday to cover for the time I lost on Monday writing Week 2, but after two days working on this until 1am, waking up early again, and dedicating all of my energy and thought to the problem, possibly doubling (if not more) the amount of time dedicated to this effort than in previous weeks, I clearly needed to lean back from the table and look at everything from afar again. Surely my mistake was again in the design. Until it came to me: the only thing I had to do, was break up the ties when the compare: function wasn’t enough. If two Tickets had the exact same time, I’d order them based on their Supplier. If two Tickets were produced at the exact same time by the same Supplier, I’d order them by User, and if the tie was still not broken, I’d use the Ticket ID, unique to every Ticket and given by each Ticket’s Supplier.

I knew this was the solution, I just knew it, from the moment it splashed into my mind. If you have been half of the time screaming at me: “don’t stop sorting using the block, just break the tie just like you said based on the idea of your Ticket ID!!!”, I agree with you, this is a very big mistake on my part. But this is exactly the reason why I don’t like being “eaten up” by a problem, because you lose focus, and you stop thinking clearly. On the other hand, it reminds you of why you do what you do.

In any case, I solved this very quickly during Q1 of the Bahraini Formula One Grand Prix. The Aggression Test was actually OK, I was just using the wrong #define values to make the calculations and was misplacing the brackets. I also relaxed the Aggression Test because it doesn’t always have to eat through all of the missing Tickets, just clear at least the same number as was recycled and not introduce duplicates into the Track. After solving that, I added a small feature I’d been needing to perform more accurate tests: an Engine restart. It does exactly what you’d presume: it returns the Engine and its Suppliers to the exact same state they’re in right after being alloc-inited. This is actually one small feature in my list for the Zealot Engine, which goes hand in hand with the cold-restart, aimed to being the closest thing to “restoring the app’s state” without deleting it and re-installing it. And of course, last but not least, I also made the switch to the Track and changed all the code accordingly. The other thing I worked on was starting to commit the code to a local Git repository just before starting my work on Gap Detection.

And with that, we’re up for a close. I hope the ride has not only been entertaining but also insightful into what goes on making an App, and remember, this is not even an App yet! If it ever becomes one, it’ll be a prettily documented one. About Week 4, I could state that my goals are Gap Filling and State-Saving. Gap Filling is in fact already done, if you think about it, since Gap Detection already inserts code into the Track, and I think I can refactor that code from the Engine and use it for Filling again, which should make the latter half of Gap Technology pretty easy to wrap up (we’ll see). State-saving is basically saving the Engine state when the app closes so the next time you open it the Track is not entirely blank, and if Gap Detection was implemented well, a new Pool should automatically spawn a new Gap. Sadly, next week I need to focus on Android for a bit, but I’ll consider either giving at least a day for this project, or posting about whatever work I do on Android. Thanks for reading, and I’ll see you soon!

Week 2 - Laying down the keel

Mon, 15 Apr 2013 12:31:00 -0400

Let’s get right to it.

Week 2 was the week where I decided the occupation that’d keep me working on iOS for an unforeseeable number of weeks, or at least that’s the idea I was pursuing. I had a rough idea at the start of what I wanted to do, but again, I did not want to mark an “X” in the calendar saying “this has do be done by then”. No. It hasn’t proven to be a good solution for me in the past, and I won’t be able to change the outcome of things if I don’t change the process itself. Yes, there were goals for week 2, but there were very reasonable goals, and there were goals I knew I could do without rushing in. So let’s dive right in.

First of all let me introduce you to the concept I want to build. You might’ve heard about a company called HootSuite: they appeared in my radar back in 2009 when I was introducing myself to the world of technology through Engadget. They made, just like so many others, a Twitter client for multiple platforms. The difference though, was that they pursued the people riding the “Social Media” wave and built a software which would allow you to cross-post to multiple platforms, most notably Facebook and Twitter, the two kings in the matter.

What HootSuite will basically do for you is show you multiple streams of information no matter the source: it can be a Twitter account’s mentions, a stream of specific Twitter accounts (list?), a Facebook feed, etc. and of course, since this is no longer 2009, it works with a lot more social platforms since then: LinkedIn, Google+ pages, Foursquare, Wordpress… and has been refined to offer more than just “information streams”: you get statistics, options to monetize, and a lot of other stuff I’m not that interested about right now.

What I’ve always admired HootSuite for, ever since 2009, is the codebase that must be capable of handling all that kind of information no matter the source. Keep in mind whatever code is at the core of the HootSuite experience cannot know about what platforms will come next, and a reasonable software engineer/architect will tell you you have to engineer it in a way that keeps the parts that change separate from the ones that don’t. You must also follow the DRY principle, which means Don’t Repeat Yourself, which in practical terms means you should never have two different pieces of code (functions/classes/namespaces depending on the size and scope of your project) that do the same thing: centralize them, make everything use the same thing. In this way you force your code to do more work, and you catch bugs a lot faster, since you exercise it a lot more. And if you need to make a change, it’s instantly updated everywhere else: no need to chase more code.

It is not a simple problem to tackle. About why am I tackling this problem, the reason is very simple: all I want to do is practically exercise my software architecting abilities, which are important no matter who interviews you or what position you’re going to work in. Going from solving “programming problems” to trying to do something simple in real life seems like a pretty good idea to me, and what better way to do it than tackle a problem I’ve had in my head for a number of years, without actually trying to solve?

About a year ago, I started trying to type this idea down as some kind of Twitter client for Windows, based on WPF. I felt energized for Windows development given the fact it’s the platform most of my friends use on a daily basis, no matter what phone they have. This meant once I had something worth showing I’d receive a lot of support to keep it going. And I was also starting to fall in love with Metro, especially after the release of Github for Windows.

As you can see, I kind of have a knack for starting many things and not finishing them. (As well as proof of my thing for password-protection!) The main idea I had in my mind was to build a basic Twitter client, which should be much easier to do than the platform-independent near-real-time core that lies at the heart of HootSuite, and evolve things from there with the gained experience, and surprise everyone by making it a multi-platform solution. I wanted to make something similar that is Tweetbot for the Mac, and perhaps ride on top of the Windows 8 wave and make a good client capable of staying in that split-screen mode. Yes there was MetroTwit at the time, but it seemed like their standard Windows client was far ahead than their 8 release, and I could capitalize on that.

A lot of things to keep inside your head, aren’t they?

I never got the so-called Twitter client to show my basic Twitter stream: none of the C# APIs and libraries seemed to work in a straightforward manner for me, and I just simply wasn’t in a mood to work through it: seemed too complicated in my head, and I needed to relax for a bit. Then photography interrupted, things changed in the Summer, and it got left there.

Back to where we are, now. We’re going to do things a lot, lot, lot more different. I’m not going to focus on making any “app” right now: at best this is just “pre-production”, not full production like I’ve tried to attempt in the past. All I want to get done now, is basically an engine or core similar to what HootSuite must be using.

I want to write a piece of code, an architecture, a set of classes and methods, capable of handling the kind of information the engine behind a traditional desktop or mobile client at HootSuite does.

(This is a great book to have, by the way)

Question number one: timeframe to deliver a basic sketch of a working architecture? I set myself to do this in two weeks. Meaning week 2 and week 3 would be fully devoted to having a basic drawing in code of how such an engine could be done. I did not pressure myself to write it down and have it working, I only wanted to “solve” the problem on paper. This is all about software architecture, after all.

So what did I do? I started writing. Got an old but gold (it’s actually made out of silver) fountain pen my Grandma gave me nearly five years ago, and set out to start writing the basic components behind all of this. To begin with, we have what I decided to call a Ticket. A Ticket is basically a unit of information: it could be a Tweet, a new Tumblr blog, a link to a new album on Facebook, a new post in an RSS Feed: any one of those. I defined an Interface (protocol in terms of Objective-C) so I could later subclass the Ticket protocol and have different kinds of Tickets depending on the information they held. There are two basic kinds of Ticket to differentiate: there are those which are plain text, like a status update or a simple tweet, and those who come with some kind of media: this could be a web link, an image, a photo album, etc. But that’s the basic differentiator between Tickets. They’re either pretty simple, mildy complicated, or pretty complicated.

Moving on, we have the User. The User is what you think it is: it’s the unit of information covering who is responsible for producing a certain Ticket. This means a Ticket has a weak link pointing to the User who created it, or at least a means to uniquely identify it. Again, the User is defined against a protocol, so we can have different kinds of Users if we need them, or if we need to hold extra information in any of them. By default, they carry the minimum amount of information: Full (real) name, username or screen name, gender if available, birthday, a link to its avatar, etc.

Next up: the Suppliers. This is where things start to get really interesting. I actually designed this the other way round, from top to bottom, but decided explaining it the way I started to implement it (bottom to top) was the right way. A Supplier is a source of information: it handles its own Tickets and Users, which means a Ticket should have some kind of information about what kind of supplier it belongs to, and every User should know to which Supplier it belongs to, again, through a weak link against it. A Supplier produces both Users and Tickets, and thus, it is the first component of our architecture that really knows which specific classes (protocol implementations) it talks to. This is important. A Supplier strongly links to its users, and forms its own “family” of products. In its own way, a Supplier is a content factory. I would not really describe it as a Factory pattern, because my idea was that the Supplier should be the most flexible component in the whole system. It’s the one that adapts depending on what kind of information it needs to bring: RSS, Tumblrs, Tweets, etc. And whom does it feed the information to? That’d be the Engine, of course.

The Engine. I’m going to define the Engine as the Data Controller. It is the closest piece of our system to the UI, and it acts as the component in charge of making sense of all the Suppliers’ information as well as the requests the user makes through the UI. If you’re familiar with Android, the Engine is the true Adapter, or provider of information. In iOS, this means the TableViewController talks to the Engine and receives its content through the Engine. Thus, the Engine manages the suppliers, notfies them when the user is requesting for new information, receives all the latest information from its Suppliers, makes sense of it, and gives it back to the TableViewController. Simple enough?

All of these concepts were pinned down by Monday evening. Having done so, I started coding it down. I decided not to worry about blocking the main thread here, leaving any kind of asynchronous work in the back of my mind for now. What I really wanted to do, was to write the basic architecture I’d just designed, and see it working. It did.

It was a very pleasant to see it working. On Tuesday I finished writing all of the code, corrected some protocol properties here and there, and got it up and running. I implemented the standard “Pull-to-refresh” API provided by the TableView (by the way, Xcode still doesn’t allow you to “link” to the Outlet in IB; you must set the selector in code), and after every refresh two new tickets were added. I wrote some Dummy implementations of the Supplier and User classes as well as the Ticket. The Engine was by far the most developed component, since it had to query all of its suppliers for real. One was enough for me, so I simply detect whether all suppliers have provided their “goods” by tracking how many have called the Engine callback block in an NSInteger __block variable. Not the best solution, I have a much better one in mind, but again, this was just a simple proof of concept: I wanted to see it working. As for the name, I called this concept “the Zealot Engine”. No class actually bears that name, although protocols start with the “ZE” moniker. As to why “Zealot”, you must’ve noticed most of the names don’t really carry a real resemblance. Tickets? Supplier? Engine? The names used in the code, the concepts guiding the development only need to mean something to me, so I pursued this line. The current Engine implementation itself is called “Standard Engine”, in honor of the “Standard Model” used in Physics and the fact the Higgs Boson or “God Particle” finally made its appearance last year without disrupting it in any way.

Before we move on, just a couple more details in the implementation side of things. The cells you see in the image look like standard “Subtitle” cells to you, and they are. The interesting part here is that I made every Ticket have a weak link to its Supplier, too. Why? Because I added these two interesting methods to the Supply protocol:

Yup. Neither the Engine nor the TableViewController displaying this kind of information is responsible for handling a Ticket’s onscreen representation. Who’s the one that really knows exactly what kind of information a Ticket carries? The one who created it, its Supplier. So what the TableViewController really has to do is grab every Ticket’s Supplier and ask it to provide him with the final UITableViewCell. Simple and elegant. The second method is for the future, when we decide to “expand” the available information and require a new UIViewController to come on screen. Who’s the one who’s going to know who is exactly the best class to display a certain Ticket’s characteristics? You got it. The Supplier.

Also, about how the Engine stores all of the Tickets, the simplest solution was an NSMutableArray. This array has a readonly property returning an NSArray, so no one can externally modify the contents of the Engine. But displaying its contents presents a problem, which is that the array itself grows adding new stuff at the end, and I want to display content from newest to oldest. To keep the array ordered in this manner, we’d have to constantly be reordering it to keep it in sync with what’s on the screen. To go around this, I inverted the way the Tickets are displayed: instead of displaying array[i] to array[[array count] - 1], what I do is display array[[array count] - i] to the end; this makes perfect sense, because adding new data to the array is best done at the end (conceptually at least), so every time we update we only add the new Tickets in oldest to newest order at the end of the array inside the Engine, and everything works.

With what I call “the basic mechanism” working done, on Wednesday I set out to improve it by adding asynchronous operation. I wanted to add sleep commands inside the supplier and see how the tableView refreshed without blocking the UI. I read a lot on the subject and watched multiple Apple sessions. In the end, I got it working with a concurrent queue GCD I create myself, and I add the barrier inside the Engine (code) block that every Supplier calls back so the critical section is mantained: this is because we don’t want multiple threads “touching” the code that adds new tickets to the Engine nor the counter keeping track of how many Suppliers have delivered. I also gave a name to all the fresh tickets being delivered by the Suppliers before they’re put in final order by the Engine and added to the internal array of tickets: a Pool.

No, I’m not talking about that kind of Pool. Nor was the pun intended. I could’ve actually called it “Basket” for example, but I thought the name Pool was very common in Computer Science literature (Thread Pools, anyone? Just like the one we’re implementing behind-the-scenes) so I went ahead with that name. So, the basic idea is: the Engine gets called to refresh and add new Tickets from the TableViewController running in the main thread: the Engine will then call every supplier to deliver new tickets asynchronously in a new concurrent thread (I decided to use Grand Central Dispatch instead NSOperations). Every time a Supplier wants to deliver new tickets, it calls the block of code the Engine gave them: this block has an asynchronous barrier making sure only one supplier is executing that code at the same time. This code checks out the supplier and adds the new tickets to the Pool being made. If all suppliers have delivered, the Pool gets processed (all Tickets are re-ordered from oldest to newest and added at the end of the Engine’s main array of Tickets) and the TableViewController gets notified. After a refresh, the new Tickets appear onscreen. Not trying to be heavy on you, just giving you a way to confirm any doubts I might’ve lead you before :)

This was finalized by Thursday. But I only had a very basic setup: every time you called for refreshing, only one Supplier was called adding two new tickets. Yes, making the Supplier take multiple seconds to deliver those tickets worked as expected: the UI was still active and the GCD code was working as expected. But I needed to really test this. I wanted a way to test the engine to the max, put it under stress, and make sure all the different parts of the code I’d made thus far were really working: the asynchronous code, having multiple suppliers, the barrier controlling access to the critical section of code, and of course making sure those threads weren’t making the Engine lose any tickets. Plus, the Tickets always need to be in the right order, from oldest to newest, from the beginning to the end of the Engine’s internal array. The answer to this was Unit Tests.

By Friday I had them working, and after correcting some math operations, it all turned all right. I made a new Supply, which I called the ProgrammableSupply, because I can change its name at runtime, the number of users it has, and the number of Tickets it produces in every new Pool. The addition of this ProgrammableSupply made me realize one thing: I don’t want any of the code in the Standard Engine to change, and yet, changing the Suppliers meant entering the Engine and changing some code. So what did I do?

At first, I thought of it as a real Factory Pattern: the Engine would be composed of a new class (or family thereof) that’d be responsible for creating and initialising all of the Suppliers in use by the Engine. The Engine itself does not need to know who its suppliers are, only their interface, which is common and provided through the Protocol we’ve spoken about before. Then I decided it’d be a new component, the Commander.

The Commander is responsible for creating and mantaining all of the Engine’s suppliers, and thus, if everything goes all right, I can build every single feature I want into the Standard Engine and keep the Suppliers completely detached from it. The Commander and the Suppliers will be the ones holding specific implementations, but that’s their job.

With everything tied, I dropped work on Friday at about 9pm. I’ve not worked as much as you’d imagine on this: it’s been a balance between giving it around 2-3 hours at the desk plus many minutes or even hours while doing other mundane tasks thinking about where to go next and what could be changed. But this is the reason why I wanted to move slowly: I want to get it right, and not find out later that I’ve been taking care of 95% of the cake but left a 5% with pretty sharp edges to deal with.

Surprisingly, instead of taking around two weeks I took just two days to have a fully working prototype, which leaves me at odds with what I’m going to be able to accomplish this week. Will it be a lot? Will it only be a little?

It’s been a very long post, so I’ll leave you to fully revise everything I’ve written for the past two days. If you allow me, in the mantime I’ll keep going. See you next week :)

Week 1 - Password Protection

Sun, 14 Apr 2013 13:53:00 -0400

OK, time to fill you in on what’s been going on.

21 days after my last post, it’s time to bring you back up to speed. I have been publishing some photography, as you’ve seen from my Twitter and Google+ pages, and even though I’d normally be posting my own words next to all of them, I haven’t: something else has been eating into my writing, which is programming.

Up until two weeks ago, I’d been heavily working on picking up my iOS skills which had been pretty much left to dust since the Summer of 2011. Back then I was pursuing a very, very big project which was completely out of scope for the time I had available it even if I was unemployed back then (the primary reason why I took the chance to follow it). Now, I know things don’t count unless you finish them, so that scar has been on me since back then: a ghost telling me there’s something I set out to do, but failed to deliver.

The specific reasons why that project failed are not for today to discuss. What you should know though, is that it’s been a weight I’ve been carrying over me for a very long, long time, and perhaps I might sought out to lift it off my back now, step by step, correcting everything I did wrong that time, and not “running into” things, but eating the cake piece by piece: starting from the outside, and carefully running to the inside from all sides, not biting from different angles to eat it from different directions managing different goals at the same time. I’m going to take this slow, but in a very methodical manner.

Two weeks ago, just as I published my words on the last Tomb Raider game, I’d also managed successfully to complete Ray Wenderlich’s excellent guide to iOS: I bought both the Apprentice Bundle and the iOS5 & 6 by tutorials. It’s not “excellent” because it succesfully targets people who haven’t programmed in their life (it ramps up far too quickly due to how iOS and Objective-C are and the example apps the book tries to make) but because it explains what it does, and it goes sufficiently in-depth. This is what I like about books like Mark Murphy’s on Android: they explain why they’re doing what they’re doing in their code. It’s not just “this is the code for doing X”. No. It’s more like “the API wants this, so we’re going to give it what it wants in manner Y”. This is essential to learning a new platform, since just copy/pasting code from StackOverflow and fighting the System’s APIs is not the way to move forward: you’ll constantly hit walls and wonder why other people don’t. It is something I miss from Assembly and the likes of pure C/C++ terminal code not based on frameworks you don’t know, but that’s just the state of modern development these days.

Moving on, after that I decided it was time to do something after having just finished the Apprentice Bundle, or else my recovered iOS skills would slowly fade away once again. That’s when I remembered one of the things I like the most when writing a new project: protecting it.

(Early image of the project I was working on in Summer 2011)

Even back in the Summer of 2011 when I had a project in front of me, the first thing I wanted to do was write some code around a basic SplitViewController to ensure any time the app would run it’d be password-protected from anyone (friends, family) who’d catch eye of “that strange icon”. And if I did it right at the beginning, password protection would be a feature I did not have to worry anymore. This made my 2011 Project turn around and put this feature first and foremost: requiring me to successfully implement password protection by tapping a button, by turning on the iPad after using the app, by using the app for the first time, and of course, being capable of both setting and unsetting the password. Last but not least, I also had to save all of this securely and behind the app’s preferences. If I were going to do something, I’d better do it right from the start.

What looked like a “basic feature”, as you can see, simply wasn’t. A lot of code had to be written to go around and build that feature, starting from the Preferences and the handling of the password’s encryption. Now, I’ve always liked Computer Security: I love the idea of hiding secrets and being able to efficiently protect them, but I’ve got to admit there’s not a lot I know about real Cryptography. I have the basic ideas planted in me, but I don’t know a whole lot. What I can tell you is, there are a couple of rules that don’t hurt when it comes to securing any kind of system:

1) NEVER keep passwords in plaintext
2) The LESS information about the user and his/her password you have, the better
3) [Computer] Security is only as good as the WEAKEST LINK in its chain

I’ll review them very briefly: you’re going to say “of course I’m never going to store the password in plaintext, that’s why we use fancy algorithms to encrypt it!”, but have you thought about how you’re going to check whether the password is correct or not? Surely you’ll decrypt it, compare it with what the user just introduced, and then verify if it was correct or not. DON’T. Simply encrypt whatever the user types, and compare that to the encrypted password, but ever touch the password without any kind of encryption.

About numbers two and three: if you can avoid storing the password and the user’s credentials, go ahead. This goes hand in hand with the third rule, which is that no matter how good the strongest part of your security system is, hackers only have to break that poor little code with the tiniest amount of responsability to break into you. I’m not personally a computer forensic nor do I have any idea of how to protect myself against each and every kind of attack our software is prone to. But the above principle remains: the least we know, and the more we can stand on the shoulders of others who know more than we do, the better. Our app will handle less confidential information, and if someone manages to break our security, we should minimize the chances of that information being of any use for them as much as we can. We are not being stupid and or “bad developers” for following this principle: we’re simply being smart about what we’re good at doing and about what we’re not. We’re acknowledging what our strengths aren’t and protecting ourselves from them, taking measures. That’s what adults do.

Back to my password code. See how the simplest of things raises so many concerns? My lack of vision is probably the single most tallest wall I hit and fell at many times two years ago, I mistake I wish not to make this time. But before thinking about making something, once again, I’d like to protect it: standing on the shoulder’s of iOS 5 ViewController Containment APIs, I’d like to make a small re-usable component I can call in from anywhere to protect a certain iPhone ViewController. And we’re going to tackle this bit by bit.

How will this new component work, exactly? The ViewController (henceforth VC) requiring protection will be responsible for asking for it, by alloc-initing my component, which will automatically insert itself in the parent (VC) View, present itself with some cool animations, and prevent the user from moving forwards until the password is introduced. When it is, it’ll make way for the VC that called it and restore the original view, successfully being dealloc’ed after removing itself from its parent’s view. From a Software Architecture point of view, this means the outside world is responsible for knowing whether it wants to be protected or not (the password protection setting). That’s OK, I only want to build the component for now and that’s where my primary focus will be.

(The Password Protection screen for my 2011 Project)

Next. We can show the Password component, and dismiss it, all with just a couple of ViewController Containment API calls in the component itself (it’s its responsibility). How de we encrypt the password once the user introduces it? Surely iOS must have some AES encryption algorithm or API? It does, but it’s not the correct solution. One simple visit to Google and a link to StackOverflow reminded me that the best way to store passwords in iOS is actually, the platform’s Keychain. It’s a common API for both (Mac) OS X and iOS, and the best part is, I didn’t have to write the code to talk to the Keychain API, which is not an Objective-C API, but classic-C based. How did I not write the code for that? Well for starters I picked it up from that 2011 project, since I remembered that was one of the very few parts that worked really well; it turned out, there was a guy I respected a lot in terms of iOS development who runs the blog Use Your Loaf, and he wrote a nice and concise piece of code to talk to the Keychain. Back then, I wrote it back myself in Xcode, copying it by hand. Why did I waste that kind of time? Because copying it by hand would give me a chance to look at it line by line and understand more or less the idea behind what that code was doing; I don’t need to be an expert of every single corner of an SDK, I only need to know how and when to use the correct parts of it, but trying to understand code that’ll be in something I’ve done never hurts.

Two problems solved. One thing now: the VC calling us knows whether it wants to be protected or not, but after that, everything password-related is our component’s responsibility Where do we store whether a password has been set or not, to know if we must ask the user to set it? Though I could rely this to another component, touching the password is our responsibility and so is both setting it and possibly unsetting it, or at the very least delete it, at the user’s request. Setting it though, should be pain-free. With a simple block callback we can notify the component’s user whether the password has been set or not so he/she can update whatever code he/she needs to. But we need a safe spot to store whether a password exists or not. We could use the Keychain for that, but I preferred to tackle the Preferences problem because a class capable of handling key/values and storing it in a safe manner is definitely handy. Especially if I can make use of it again. But how? Subclassing NSMutableDictionary? A new class composed with an NSMutableDictionary? I opted for the latter, and used the iOS File Protection API after carefully examining this Ray Wenderlich tutorial among others. Although the File Protection API only works if the user sets a passcode on their device (it works through hardware encryption), it’s a good hassle-free solution, relatively easy, short, and can work just as an NSMutableDictionary adding the proper methods. We only have to worry about the data protection when loading the Preferences from the file and when saving it. Done.

With these pieces on hand, the only remaining bits to complete the set where the animations and a little bit of refinement. I did not give the end result some nice colors and textures nor did I write an API to unset the password: I just wanted a couple of features and I wanted to set myself a very reasonable goal as a first step. It was the first time I was doing something in iOS in two years, and I wanted to start well, fresh, energized, and with a small success to lean on, and thankfully I did.

UIView animations turned out to be a little pesky, since they don’t necesarily work “linearly” and instead opt for lazyness, taking “the shortest route” to move Views. I also found out Auto-layout was screwing any type of animation I was working on, so I set it off in the Password component (the ViewController’s own nib): I did not want to bother either about the larger 4-inch screen or landscape mode. Not today.

What I did work on is in notifying my test (subject) VC of when the app got suspended and/or resumed: this is what eventually sets off the Password ViewController. A much, much better solution than inserting code in the AppDelegate, if I must say.

(The real deal running in the Simulator)

All in all, I can admit I’m pretty satisfied with the results. Granted it was not much that amount of work since some of it had been done before, but the thing I liked the most was that I did make a re-usable Password Protection screen (which can be expanded for iPads, too), that it works in a respectful manner with Apple’s iOS APIs and Objective-C, and that I hit my goal by Friday afternoon just before lunch. Sweet!

Tomorrow marks the start of week 3, so as you can imagine, week 2 is where I start setting up goals leading to something more than just a re-usable iPhone ViewController. I’d love to get it done tomorrow itself, but if I do that’ll be less time during the week to actually do some progress: this is the main reason why I’ve been posting pictures and not writing about them. I prefer to spend time working on this rather than dedicating 1-2 hours to writing. But I understand in this case it’s of extreme importance to get it done, so I think I’ll make an exception for tomorrow, and get you week number two.

By the way, please feel free to comment or provide your own opinions on my work :)

(Note: You’ll notice the cover picture for this post shows a “Lock” button on the upper right corner. I removed it after detecting when the app resumed worked to lock the screen, and kept an older build in my phone for the shot.)

Lara Croft's Tomb Raider (2013) is born

Sun, 24 Mar 2013 09:19:09 -0400

I’m honestly having a hard time just coming up with a title for this post. Wait, I just figured it out.

Um. Honestly, I’m not really sure about where to start. My history with the Tomb Raider series began somewhere around 1999/2000 when Tomb Raider III came as part of one of those “magic CDs” filled with games of the Win98 era. Lara could gasp, shoot, kill and even dodge incoming boulders, but my (pirated) copy of the game had no music nor voices for Lara. I was also horrible at it. I played it again, and again, trying to solve it on my own to no avail. I did the third level with the bike many times, but I wasn’t really good. Just felt frustrated playing a game I liked very much but that I couldn’t really do much about.

I would later play Tomb Raider IV somewhere in 2001/2002/2003, finishing the intro level focused on a young Lara, and when that was over, I hit a difficulty wall again.

During that time, I used to think sometimes about how Lara was constantly in the crosshairs for being something “spoling” the minds of boys who’d later look for the exact same impossible body shapes in their lives. Even as a kid, it seemed like quite a joke to me. I mean, you’d look at her, or it since she barely resembled a real girl, and clearly see that was an impossible body. It’s a nice idea to play around with in a game, but it’s no where real. So no, that wasn’t in any way spoiling my head. Not mine, at least.

As the years went through, I heard about what happened to Core Design, how Crystal Dynamics wanted to re-energize Lara and had brought her back with Legend but didn’t quite make it with Underworld. Legend was OK to put Lara back in her feet, but Underworld was not taking the franchise anywhere you’d want to go. Or spend time playing Lara.

So I lost track of Lara, mainly. I just didn’t like what she’d become in my mind. She stood like an impossible body, composed of legs, limbs, a torso and most importantly a face and expression in which I didn’t see anything I liked. The whole idea of that Lara just wasn’t appealing to me anymore, and I didn’t understand how I’d spent so much time with her in my childhood trying to solve those puzzles in the jungle. And the city (via cheats) because I loved her night outfit.

I wouldn’t go as far as calling it “hate”, but she was simply empty in my eyes. A shell from an old era I thought did not have a reason to exist. There was no Lara Croft really, just another “Tomb Raider” game in which you played the same character, or seemingly did, and did the same stuff as before.

Then the fresh waters hit in June 2011, and I just couldn’t have enough of this new Lara. It seemed like this was the real Lara Croft, not the other one. What was the other one, now that I came to think about it? A shell composed of impossible jumps shooting ahead wherever she went to get what she wanted?

I went so crazy after news of this new Lara hit, I revisited her entire story reading the plot behind every game in Wikipedia, trying to get a feel of what had really been going on with the character behind that old face.

Beyond what words could say though, this was a Lara I really wanted in my life, or at least a new character I felt deserved a chance, and I wasn’t wrong in any way.

Back to the game. I started playing it last Saturday, mere hours after finishing Heart of the Swarm. I played about 3-4 hours to fill my all-nighter into the first round of the 2013 Formula One Championship, and I’ve got to say picking up the game so quickly after finishing HoTS was a mistake. I practically went through those hours because I realized the game was keeping me awake and I knew if I tried to do something else sleep would catch up with me. More than a couple of times through those hours I felt like stopping and putting it down, almost not feeling a reason why I should really continue playing it.

I ended up having fun, until I hit a roadblock and decided it was time to watch the race. Then I slept, had lunch, wrote the piece on Starcraft II’s first expansion, and just couldn’t stop thinking about anything else than Tomb Raider. I didn’t feel as connected to it as I did to Starcraft II, but we were in the month of relentless gaming (which is now over), and I really wanted to see it through, so I played about 2-3 more hours that Sunday, after hitting another roadblock and wondering whether I’d be able to continue through the adventure.

The next week began, and through the majority of the days I wasn’t actually feeling too good for the first time in months. A small ray of light hit me on Thursday evening when I suddenly thought to myself: “what if you consider you’ve done enough for today and get back to Tomb Raider?”

That’s when I really started enjoying this game.

Lara became herself. I started understanding her, I started understanding what the game was all about, I started understanding why I had to do everything the game asked me to do (saving my friends and getting out of there), and I really started to understand the mechanics, the skill points, and how trying to kill everyone by a direct 1-shot to their skulls made me reminisce of the old Counter-Strike days in a way I found deeply enjoyable.

Come to think about it, it’s all a mix of many well thought things that make this new Tomb Raider what it is, most important of all the protagonist Lara Croft. I feel the game is called Tomb Raider just as a service to the original titles, not because it should really have that name. This Lara is not a Tomb Raider; this is the real Lara Croft. The Lara with a personality, with a heart, the Lara who despite being surreally gorgeous (more on that later) has aspirations far from enjoying her money and making use of any quality time she can steal from good looking boys on sight. In one of the game’s diaries, Sam even suggests she had to drag Lara to meet cute guys… that’s a depth I’m more familiar with in TV Shows, and all I can think about is that we have Rhianna Pratchett to thank for all of this.

I mean, this game is SO GOOD. It feels so complete, there are seemingly no holes, it’s easy and challenging at the same time, you grow to care and protect the character, and the finale is truly incredible. It seems like this game has been thought about tirelessly again and again, making sure every detail was looked after to make sure the package became a truly cohesive experience. I can really find no “buts” overall. Camilla is fantastic as Lara; if there is a new film (I wish there wasn’t), I’d go as far as suggesting Camilla should dub whatever actress lands the role.

Part of what makes this game so good is the feeling that the ending is really open. About halfway through the game, I really thought it’d ended!! And to my great surprise, it kept going on, and on, and on. Total playtime for me had to be about 12-14 hours, and the last 3-4 happened yesterday (Saturday) morning after the 2013 Malaysian Grand Prix Qualifying session. I just couldn’t get up from my seat. And in the evening I began reading into the character’s story to write this piece. By the way, the retrospective I liked the most was this one, although it lacks the “Underworld” part.

But now it’s time to enter the only “but” I have against this game. Not necessarily about the game itself, but around Lara’s character design.

There’s no denying they had to make the girl insanely beautiful, and I agree with that. To me, I like her the most from the CG renders, and her in-game looks lose against those of Starcraft II’s Sarah Kerrigan. I’m all face & eyes when it comes to girls, or “features” as my Grandma’d put it. Maybe my i7-920, 8GB 1033 RAM and GTX260 didn’t allow me to contemplate Lara’s features better, but I think her face is a bit different than the one seen in merchandise, artwork and CG trailer, and to me Sarah’s face looked better inside the game.

With that behind, I’d like to concentrate on another matter, her body constitution. This new Lara is definitely shaped to be real, I mean, she’s not very tall, she looks healthily thin, her clothes aren’t exactly high-fashion, her hair is tied in a very simple ponytail; these are all details looking to providing the image of a Lara who’s rather a very approachable girl. You get the feeling a girl you might look someday walking down the street could look as good as her if not better. Perfect. I agree. Trying to provide a Lara closer to reality doesn’t mean her body can’t be gorgeously shaped.

But this is were I started to figure out a big detail.

The game is constantly trying to make you realize how generous Crystal Dynamics was with new Lara’s breast size. And by that I mean shots like this, where it’s seemingly clear she’s actually not so real. Girls do not need a huge breast size to be stunning, just look at all those VS top-models earning hundreds of thousands of dollars an hour who aren’t so gifted as Lara in that sense. I don’t quite understand why she had to be given a big cup size if they wanted to make her look more real. I understand, perhaps, that it might’ve happened because they simply wanted to emphasize how young and stunning Lara should be, and since her clothes don’t play so much to those strengths, in situations like the one below they wanted to make sure she she still looked like a really attractive woman, and in camera situations like these in which the game so often finds itself, well, you just needed to give her a certain breast size in order to make her figure as noticeable as they wanted it to be in the dark.

OK. That, however, doesn’t explain why when it comes to first person shots of Lara (and by “shots” I mean in the sense of photography), the camera is constantly pointing out how big her breasts are. Look no further than this sequence:

(Sorry about the first picture, I play with subtitles turned on, even though I write, speak and understand English perfectly. I do the same with films.)

C’mon, tell me it’s just not me. You could argue, if you’re the DP (Director of Photography) because there just has to be one in such a big budget game whose scenery easily produces are stunningly gorgeous shots, that this sequence is simply trying to emphasize on what Lara finds inside the sword. Right! If you’re a DP, you could also think about it in terms of negative space and then come back to me and tell me what else is there in that scene to look at that’s not Lara’s um, well I’m tired of saying it.

Of course, you could say me: “If I were DP I could do a lot worse to get you to look at Lara that way”. Oh, you mean like this?

I’m going put an to end the argument here. I’m really not against Lara being beautiful (face), gorgeous (body shape), I’m simply pointing out the camera isn’t shy of the fact she’s not as real as she seems to be. Not all girls could look as good as she does naturally. In the battle of sexes, a lot of guys are portrayed as big and muscular. Now, height is not something we can do much about, but getting in shape is feasible although it might take years; my argument here being if a guy wants to be in shape and enjoy a good-looking muscle structurer, he just has to put in the effort. To girls we’re simply telling them “you can get your body to produce a healthily nice figure through hard work, but you either have big breasts or you don’t”. As long as nobody here understands we are punishing women for not looking like Lara, that’s fine by me.

The matter is not over, though. If I thought Lara’s breasts in the new game were a matter worthy of a couple of paragraphs, you can imagine my shock when in one of the many retrospectives I saw on YouTube I noticed this not-so-subtle-effect from Underworld.

I mean, WHAT ON EARTH WHERE YOU THINKING, CRYSTAL DYNAMICS!? No wonder the series had fallen into a decay under general perception whilst being tagged as a male titillator. That’s clearly not realistic and not in the tone of what I thought Crystal Dynamics wanted to do with Lara when they picked up the series. And then you find Lara dressed in costumes like this, seemingly ready to fight and shoot with any or all of her weapons in her arsenal:

The idea is simply disgusting. But maybe, maybe this speaks even more about what Crystal Dynamics has actually achieved with this new Lara.

She’s not only human, she’s also caring, and most importantly of all, she made me care for the story of how she picked up her first gun and delved into a new world for her.

The game is so great, I cannot insist in what I said before: this is not about the Tomb Raider of the past, this is about Lara Croft. And keeping in mind how reboots are going lately, I wish a “Dark Knight” awaits for us in the future rather than a “Quantum of Solace”. Of course, if we do end up in a couple of years with a QoS, it’ll hopefully be followed by a “Skyfall”.

In short, Crystal Dynamics has given a new life to a character belonging solely in the past of video game history. Personally, I’d rather forget the old Lara existed and just remember that what we have now is an amazing character who discovers herself wrapped in an amazing story as part of the great package that this new Tomb Raider game is.

I don’t know how soon it’ll be until the next chapter in this amazing reboot arrives, but there’s one thing I know for sure: not matter what happens in the future, I’ll always have this Lara to come back. The Lara Croft I chose to remember, the one I choose to keep next to me, forever.

El Teide under the snow

Sat, 23 Mar 2013 15:30:07 -0400

I’m currently more or less under a strict schedule when it comes to my time. This means I dedicate mostly weekdays to push forward on my aspirations such as (fully) picking up iOS development again and becoming comfortable with programming interview questions. But, there are a couple of things I did allow myself to creep into that schedule, like photography, driving, and photography. Not necessarily in that order.

The last time one of the most iconic views of the Canary Islands and Spain in general got completely covered in snow was in February of 2012. I was here with you at the time, but not into photography in DSLR format. Snow came again this February, but my schedule wasn’t exactly ready to take a 1-day hit like that, so I stayed at home. However, this very same month (yeah… like nothing’s been happening this month, and more) it happened again. And even though I was promised it’d happen again, I didn’t take the chance and went.

That happened on a Wednesday. I came home that very same day, having driven many hours and taking pictures to more things than just the white mountain (the Mercedes SLR McLaren in daylight). The feeling I had is that people (namely, my friends) might want to see those pictures. So I went ahead and on Wednesday night quickly (er… 30 minutes minimum) edited one and posted it. Success.

I got a little bit hooked, if I can admit that. So I kept going the next day. There was actually a picture I really, really wanted to do; it even made me stop at a spot I wasn’t really expecting to do so and in consequence took this picture almost by accident:

Funnily enough, I describe it as an “accident” because I was forced to take that angle because I wanted to portray the snow and because right underneath where the frame ends there were a lot of people goofing around with some remnants of snow. Now that I think about it, shooting that wouldn’t have hurt either. Anyway, at this point I’d taken out my 50mm f1.4 and swapped in the classic 18-135mm f3.5-5.6 kit lens, which although we all agree isn’t as good looking and feeling as my prime, it’s quite saved by the marvels of HDR photography bringing out details we couldn’t even notice were there.

Yup. Like how that one turned out. However, it all isn’t as perfect as it seems: the image is actually a bit blurry when you zoom into the rocks. Why? Because the wind was so strong my aluminium tripod couldn’t stand still enough.

Nevertheless, I know present you the picture I really wanted to take:

That’s the view I really, really, really liked. However, I just didn’t feel it when it came to editing it. It just didn’t seem “fulfilling enough”. It’s not reasonable and it’s not entirely logical, but I didn’t like going down that road. The view of the clouds crashing into that snowed ladder needed a little bit more context. So I picked up a wider view I had, showcasing much better the impact of the snow with the Sun falling down. It almost seems like Mars:

Yup, that’s one I really like.

There were more, of course. Many more. However, keeping up with the previous three days, I had to focus on a picture I really liked or I wouldn’t have been motivated enough to continue, so I just focused on that picture of Mount Fuji (or similar) that Apple has in OS X with the mountain covered in snow and lit pink by the falling Sun, and took one I thought would look spectacular and set out to achieve something similar. Here it is:

It was not HDR’ed, contrary to the previous three. There are, of course, many filters and effects applied, and although I’m happy with the upper and bottom portions of the screen, I’m not with the middle, which got completely lost (it’s all pitch black) to provide that effect. Perhaps I should’ve worked on an HDR version of the view (I did get all the exposures) but, what’s done is done, and I prefer to keep it as a reminder of a mistake I made.

There’s just so many pictures I could edit… let’s get to it slowly.

About the Polytron Machine and the Endianness Problem

Tue, 19 Mar 2013 20:08:00 -0400

This is going to be a bit of a long introduction, so please bear with me. While following tutorial 3.5 of Ray Wenderlich’s wonderful iOS tutorials, I hit a point where I needed images in my iOS Simulator. Easiest solution? Open Safari in the Simulator and start “saving images”. To accomplish that, I hit one of my daily sites, Ars Technica, and stumbled upon a recent article on Fez. I did get a picture out of that article, but the best part was really watching the video. Please do it; I cannot spoil it for you.

I mean… it’s REALLY cool!!!

Spoiler alert if you haven’t seen it yet!!!

I mean… when I read about the “reboot” thing I just had to check it out for myself, even if I was busy actually doing some coding and have a strict policy for not falling off into distractions, which can easily break your “flow of work” and kill the chance of you being able to do anything productive.

But it’s SO WELL DONE. I mean, you start seeing those glitches in the screen going on so quickly and you think “yeah, it’s the game, it’s just a trick”. But suddenly everything comes to a halt, and the sound gets “stuck” just like it used to do in the old Sony PlayStation when the game crashed, or even in my old PC when a game died mid-session in Windows leaving the Redmond-born OS recovering painfully slowly, with the screen recovering first, and the sound later.

Honestly, though? What really made put up Fez as a statement of how games carry through the voice and personality of their creators is what comes right next… the Polytron Machine!!!

Wow. That almost took me back to the days of DOS. Seriously, 640Kb of memory? And a 23Mhz CPU? Wonder what architecture it’s running on. “press Alt+F2 to enter FISH utility”. This guy is a legend.

After that, Fez turns into the 2D/3D hybrid it’s known for, the one I discovered in a little film called Indie Game: The Movie; film which, if I’m honest with you, I regret not putting my own money into when I first heard about a thing called “Kickstarter”, and which I have a new desire to re-watch in its entirety.

Let’s just hope it comes to the Mac very quickly, please.

Onto the second half of this post: endianness.

The last problem of a well-known book desired to get you as prepared as possible for a tech programming interview is about a quality hidden inside all modern CPUs which ends up not being very important nowadays (in fact, you could completely forget such a thing exists and have a successful career as a developer in your specific area/field of work), but which turns out to be a pretty interesting question for potential candidates of top-tier tech companies as it involves a very simple problem requiring the use of basic boolean arithmetic and making use of logic; no specific frameworks or languages involved, just plain problem-solving using the foundations of what powers Computer Science. First though, I’ll answer the question: “what is the endianness of a CPU?”

In short, it’s very simple. It reflects how a CPU writes numbers: whether it’s left to right (BIG-endian) or right to left (little-endian). And yes, it only and exclusively relates to numbers, but because numbers are pretty much everything when it comes to assembly code, the one most directly understood by a CPU, it lies at the heart of potential problems when different CPUs with different endianness interact. After all, if I read and write left to right, and you read & write right to left, even though we’re both speaking in English, none of us will understand each other; we need to translate the other’s writing (transpose, to be precise) first.

This simple diagram from Wikipedia might help:

My point being? Back to the book, I obviously tried to write a solution on my own, before hitting a flawed assumption which didn’t allow me to find it. Out of ideas, I looked at the answer and found out it was the one I’d thought about before my flawed assumption stepped forward. The code in question is very similar to this one, (if not the same, I just wrote it on my own for testing) since the basic strategy is correct:

As you can see, we simply put a “1” in a register/memory address, get an address to its lowest byte (there’s always a first and a last word in a sentence, no matter in which direction you write) and then perform a bit-wise AND operation comparing the lowest byte with another “1”.

At first, I thought this code would end up being wrong, because I thought it relied on the fact the second “one” would be AND’ed in a different way as the first one to detect whether the first bit (there’s always a first character to every word) is a zero or not (leave what that means for later). Why? Because I thought that code led to comparing “10000000” in the BIG-endian case and “00000001” with “00000001” in the little-endian case. It doesn’t, of course. I tested what happened if you compared “00000001” with “00000001” and “10000000” with “10000000” and sure enough, there’s also a difference in the first bit (lowest address) of the number resulting from the AND operation. Might sound stupid now, but it really wouldn’t when your mind moves slower than a turtle during a live interview. By the way, it’s best if you do this yourself on paper; it’ll become clear.

The meaning? If both first bits are “1”s (AND operation is true), then it is a little-endian machine; if both are zero, (AND operation is false), it’s a BIG-endian machine.

(If you too have trouble remembering which is which sometimes, use this: little-endian means the least significant bit is in the least-highest memory address).

I was actually hoping to use this code for comparison between maybe my (x86) Mac, the iOS Simulator (runs x86 code), my iPhone (ARM) and an Android emulator (translates ARM instructions to x86 for execution). The problem? Even though ARM chips started being BIG-endian, they’re actually bi-endian these days, meaning they can switch from being little-endian to big-endian at our command. In practice though, Android, iOS and WP8 devices all use little-endian as their form of numerical representation, so that’s off.

But that doesn’t mean there aren’t interesting questions to ask. For example, is there any other way of solving this? How about doing this in Java?

Before actually checking again the book’s solution worked, another (easy) way became clear to me: set a variable to “1”, and perform a left shift by one bit. It’s a left shift, so no sign extension is involved, and the solution is pretty obvious. “10000000” becomes absolute zero, and “00000001” becomes “00000010”, so a simple test of “bigger than zero” answers the question of whether the CPU is little-endian or not. And you cannot point to an integer in the stack in Java, since references only exist with Object(s), so that’s a clean (and perhaps original) way to answer that question.

Then one last question came to mind: “both Intel and AMD CPUs understand the same set of instructions. Supposing AMDs were BIG-endian, could code compiled for one be used in the other and vice versa?”

Yeah… stupid and at the same time a bit far-fetched. But I answered my own question, and it’s a “no”, and the answer is simple: “00000001” is not the same instruction as “10000000” (yes, I know that in two’s complement the latter is a negative number, but it’s a pointless fact here), so the resulting program would not be the same “program” if little-endian code were executed in a BIG-endian machine or vice versa. Think of it this way: the sequence of numbers “1 2 3 4 5” is not the same as “5 4 3 2 1”; even though they’re the exact same numbers written in a different way, they have totally different meaning. One is “numbers 1 to 5 in ascending order”, and the latter happens to be its exact opposite. I guarantee you instruction “1000000” has nothing to do with “000000001” in an 8-bit CPU, and much less in a modern ARM or x86 chip.

Had enough? I think so :) I just wanted to try something I’ve been telling myself for a few months, which is including “developer stuff” in my blog. Hope you had some fun, and please feel welcome to correct me if you think I’ve said something wrong!

Thoughts on Starcraft II: Heart of the Swarm

Sun, 17 Mar 2013 13:57:00 -0400

The day finally arrived, and passed before I could even realize it was all over again. Last weekend I forced myself into doing studying extra in order to do something I’d never done before: hit a game the second it launched. And the game worthy of such an honor? It could only be Starcraft II’s first expansion.

Unlike its predecessor, I started playing the game mere seconds after it’d come online, past 11pm of my Monday, March 11th 2013. It just seemed so unreal; I could not believe I was really playing the game, that the story was unraveling right in front of me, and that just as I wanted, I stayed as far away as I could from the “Hollywood trailer” which I had the pleasure to see after going through the game’s credits. And sure enough, it was full of spoilers, containing images from the last cutscene of the game. Glad I kept myself from that. In Blizzard’s defense though, it was a trailer worthy of a Hollywood film. It really was. Or is. But I wanted the complete “Starcraft” experience, just like I’d experienced it back in 1999, 2000 and 2001 when the original game and expansion became part of the fabric of my existence; being one of the first games I overcame my cheating addiction and focused on getting through the old way: by learning and adapting my playstyle. (I did not reach the end of the original game without cheats and cheated my invincibility all the way through the Brood War expansion. Oops.) But it was more than that.

The story, the characters, the drama, the universe. It was just so perfect. It wrapped everything up, and it left no holes nor ties unattended. It seemed so orchestrated, so “planned” yet undiscovered. It was the perfect blend of being surrounded by an environment you enjoy to the very last drop, the space opera, but at the same time, immersed in a set of conditions where you really couldn’t predict what would happen next: you sat there and were simply thrown from side to side as the story built itself around its characters and about all of those things lurking in the shadows you just wish you could know more about.

Starcraft, as you know, was partly “abandoned”. And I say partly because it really wasn’t; the team behind Warcraft, Warcraft II: Orcs & Humans made Starcraft, Starcraft: Brood War, and then followed with Warcraft III: Reign of Chaos and its expansion, The Frozen Throne. Then, the team went silent, and we didn’t know much about them. Blizzard became the “Apple” of the gaming community harnessing all the money in a business no one had ever really understood, MMOs, and this great franchise called “Starcraft” was left behind with unsuccessful stories like Starcraft: Ghost, which did result in a worthy character for the canon: Nova Terra (who’s a total b****, by the way!!! Play HoTS first if you want to know why). It was in 2007 when Blizzard revealed the truth: the team behind their successful RPGs had begun working on SCII right after The Frozen Throne, but got delayed by 1-2 years to help the WoW team get their money-making machine out of the door. April/May 2007. Until July 2010 the game wouldn’t hit the shelves. And I played it on the following month of December.

11:01pm of March 11th, 2013: Heart of the Swarm arrived. And it just couldn’t have been better. And the best way of looking at it is by focusing on the first campaign of the Starcraft II arc: Wings of Liberty.

Wings of Liberty brought the Starcraft world back into our era. Its addictive gameplay, where as ridiculous as it sounds, you just can’t have enough of the “build your structures, ramp up the economy, get an army, crush your rival” formula. On paper it might just look like doing the same again and again, but it just isn’t, and you’re constantly anticipating when will the next mission be. The game itself, its campaign, transported into our era the story, the world, the universe I’ve loved and kept close to my chest, making it as real as it’s ever going to get. And focusing back to the story and the game’s mechanics to push it forward, I always felt like the story’d been a bit held back. Maybe it was the fact that I already knew what the game was about and had seen the “Hollywood trailer” of the title, but the sequences, the conversations and the scenes always felt a bit “honed in”. Don’t get me wrong: it all fits nicely together from the outside, but it just wasn’t as smooth as you’d like it to be, as you know it could be, while playing it through. But the story did move forwards, only not as much as I’d hope. The game itself is insane fun, since you can’t stop doing mission after mission because you don’t really know what it’s going to be about until you actually click on it and it starts. You might be in the midst of a story arc, but you don’t really know what the action’s going to be until it starts, and Blizzard hand-crafted a game that really doesn’t make it feel like the formula is repeated or like the playthrough is excessively linear, because there’s just so many things to do, and everyone plays differently. Plus, we were the terrans!!

All in all, the story was good, it had an incredible ending, one that took us back to the drawing board saying, once again (like at the end of Brood War): “now what?”, but I felt like more things could’ve happened in the same amount of time / missions.

Enter Heart of the Swarm, where the addictive formula is kept in every corner of the title’s gameplay, and where in the campaign mode, story is everything. Right from the start, where I couldn’t help but say: “oh, Blizzard. You’re just too good.” at the so-called “Prologue” teaching you how to play the most disgusting race ever, the Zerg (really; I could swear as a kid looking at their structures evolve I felt like vomiting), the game and its playthrough are all about rightfully serving the story, and it does advance at a pace I’m a lot more comfortable with. It’s like going from visiting planet to planet and just sitting in the Hyperion’s bridge to swimming on the Spawning Pool of every Hive cluster you build (nice analogy, right?). The game just delivers sooo much more, and when it comes to cinematics, I honestly couldn’t really make out the difference sometimes. And when it comes to count, HoTS just blows WoL off the water then it comes to real CG. I don’t have a rea estimate, but WoL only had three pre-rendered cinematic sequences not counting the intro, and HoTS has a lot more, just as it should.

I was beginning to wonder whether they’d become just too difficult (or costly) for Blizzard to make them. But thankfully Blizz let all my doubts to rest with this title.

In short: the campaign is insanely good. I never thought I’d have this much fun playing the Zerg; in fact, I had doubts I could even fathom getting through the whole game playing my least favourite race, but Blizzard gave me more than enough reasons to do so, and I just couldn’t have enough of it. And like all good things, it came to an end yesterday.

Into the next topic: reviews. While YouTube loaded every available video of the Behind the Scenes DVD bundled with the physical Collector’s Edition of the game, I started going through all the available reviews (just like with Doom 3: BFG Edition, Polygon has not written one. I wonder why?) and spotted a common trend: HoTS is being considered worse than WoL. The main concern with the story is that it follows many clichés, which in some ways is true: it’s probably the first bit of Starcraft canon I’ve encountered which doesn’t totally shatter the rules of everything I’ve read before, but somehow, I’m not mad at Blizz, because it all simply fits. Critics keep that Wings of Liberty was simply too good of a game, that they enjoyed their campaign far more, and that HoTS tries to be a soap opera fuelling a love interest (Jim - Sara) with no grounds. Well, I just can’t be on your side.

(Spoiler alert!: some plot details follow)

To begin with, I was also taken by surprise at Jim’s love for Sara at the game’s opening, since we hadn’t ever seen it before. But at the same time, this seed had been planted for us in the conversation of the Battle for New Gettysburg, where “Jimmy had to drop the knight-in-shining armor”, and then in distinct parts of the story where Sara called to Jim to rescue her while inside the Chrysalis, when the Queen of Blades never even thought about killing Raynor after finishing off both General Duke and Fenix once Korhal fell once again in Arcturus’ hands (yup, this is a pretty big story) after the UED took it from him, and again during Wings of Liberty where we could see how Jim couldn’t stop feeling the blame for all of Kerrigan’s actions after that dreadful day in which Arcturus “signaled the fleet out of orbit”. Granted, Kerrigan does come at us in a pretty fierceful way toward the end of WoL, but it could be argumented that she was only protecting herself and the Swarm. She didn’t really ever threat Jim in person.

No. To me, there’s clearly a lot of story not told here. And what if these two where always in love, behind the scenes, but we just didn’t know about it until HoTS made it pretty clear? This is just a clear example of how some critics never gave the game a chance.

Wait. There’s more. Bad dialogue!? Are you kidding me !!?? Robert Clotworthy, James Harper and Tricia Helfer are incredible in their voice acting, not to mention the incredible range of Zerg monsters with their distinct habits. The most annoying creature for me was Dehaka, because he spoke far too slowly, and I like to play with subtitles on. (Any problem?) I loved Abathur, though. And yes, I also thought about his similarity to speaking lady-parts, but mostly succeeded at thinking of him as a vile Zerg creature who snapped Kerrigan’s arms and limbs to make her more “effective” during her careful growth inside the (first) Overmind’s Chrysalis.

Speaking of Overmind(s): I really misunderstood the game and thought Amon was the first Overmind. He’s clearly not. But I can’t make out if he’s the “Dark Voice” or not. Is he?

One last thing. Kerrigan is absolutely, insanely beautiful. She’s incredibly gorgeous. I really don’t know how the Starcraft II game engine made her look this good, but she beats Lara Croft in the new Tomb Raider, which I’ll get back to as soon as I finish writing this piece. Month of relentless gaming, remember?

And just like that, here we are. Back at the end of the road. For me, Starcraft is not “just a game”; it’s something that goes far beyond that, and I can’t fully explain it in words. It’s been a sort of anchor in my life, and every time a new one comes out (namely, these last two times), I tend to look back and use it as a reference of my own personal evolution, and look at how my life and my own self has changed ever since the last time it came out. I understand that for most people it won’t be the same, but I feel like this when it comes to Starcraft. I wish I could continue the story right now and keep going on, finding out what’s Zeratul’s take on the story, and what happened with that Zerg (Alien!) infested Zerg Protoss ship which we last saw en route to Shakuras, and how the Protoss government judges his actions, but a part of me knows that’s not what I really want. What I really want is start distancing my own self from the current one that has just lived through this first expansion and set the bar own higher for when I look at myself again upon the arrival of Legacy of the Void.

By the way, if you ask me, don’t expect it until the 2015 Christmas season, but I’m more inclined towards a summer 2016 release of the final chapter in the Starcraft II story arc.

For sure, the game won’t abandon me completely, having discovered Husky as my favorite caster in YouTube since 2011, but now it’s not the time to look back and marvel at all the delicious feelings this story and characters take me through. It’s time to build the next segment of the highway to my dreams; the one I only have time to build until Legacy of the Void comes around.

The Closeness of Social Media

Fri, 08 Mar 2013 09:01:58 -0500

Up to this point you must be like… “OMG! Where has this guy gone again!?”

Well, I’m not gone, not entirely anyways. Still trying to move forward with publishing a lot more about my photography, and I’ve got a lot of interesting things coming your way, but for now, I’ll keep posting final images to both Twitter and Google+ and I’ll hopefully come here and as always give you a much better view about what goes on in the sidelines.

As you know though, this has been the week of the new Tomb Raider. I’m heavily anxious about the new Starcraft 2 expansion though, and this one comes right after. But, that doesn’t mean I haven’t been paying attention to all things being said about this game; after all, I’ve been following it literally since day one. What surprised me though, is the actress behind Lara Croft herself, Camilla Luddington.

You see, the end of 2012 and the beginning of 2013 has been full of remarks about whether technology companies investing in celebrities really pays off or not; you’ve got the case of Oprah Winfrey “defending” Microsoft whilst tweeting from an iPad, “Call Me Maybe” star Carly Rae Jepsen use a Lumia 920 on her videos but using an iPhone for her day-to-day tweets… There are many examples, these are just coming off the top of my head. So when I encountered Miss Luddington’s (who’s absolutely gorgeous, btw!) Twitter timeline for the release of the new Tomb Raider, I was stunned at her campaign.

It has always been said that Social Media is not about pretending to be someone you’re not, because like in real life, it comes back to bite you, and because people already know what sort of “character” you’re playing, t’s boring, and they know what to expect. Instead, the right way to do it is to just be you; as simple as that. You must express all of the quirkyness you might have inside you, because it makes you unique, it makes your personality transcend barriers, it makes people “connect” to you, making them feel like they’re next to you, even when they’re not.

This image here, represents exactly that. Camilla Luddington portrays Joanna Wilson in the renowned series Grey’s Anatomy, and by showing us pictures like this one, she turns reality of a day-to-day experience into a window we can all look into and get a breath of what her life looks like. She’s watching someone play her game dressed as Joanna Wilson in the Grey’s Anatomy set, and she posts it like a normal person just enjoying her job. The customized XBox and the game are clearly visible, granted, but the image isn’t built from the ground up in a studio; it’s just her, sharing a moment.

So many times you look through the eyes of a famous celebrity you get the chance of putting in your living room for about 30 hours a year and you always wonder whether they’re real or not, whether they’re “people” or not, and breaking the barrier with informal moments like this is just how you get the general audience to feel like something’s real; whether it’s a brand, a game, a person, a set or a TV Show. She just nailed how to promote herself in a very natural and believable way.

Thankfully, those two were not the only “gifts” she’s left on Twitter:

She is not selling b***shit about the game, she’s just acting like any normal 20+ girl would do upon receiving a set of games and controllers after three years of hard work: taking a picture of her prize (making all the mistakes a normal person would, not noticing her own reflection in the mirror) and letting all of her “gals” (in this case, Twitter followers) know about it. She even has a convincing image of a coworker about to fall off her chair while playing the game.

Really, if she’s not the personality behind all of this, someone clearly is, and whomever you are, you’re doing a hell of a good job, and the whole social media industry should watch how it’s really done.

God bless you, Twitter. Finally they’re starting to really get you. Now let me get back at ogling this gorgeous brit until Heart of the Swarm is released.

Note: the source of all these images is Camilla Luddington’s own Twitter profile.

Game Engine Ok, folks. I’ve got to admit my own share of...

Wed, 27 Feb 2013 17:09:00 -0500

Game Engine

Ok, folks. I’ve got to admit my own share of fault regarding “letting you a bit down lately”. I’ve barely worked on my pending photography work, trying to split my focus every weekday in both re-mastering (to some extent) the nuts and bolts of iOS and in studying some books, like this one or currently this other one.

On my weekends, though, I’ve been going off anything that’s been anything similar to “work”, and I have to admit photography is in some ways “work” for me due to the level I try to push myself forward to (of course; you’re the ultimate judge in deciding whether I’m being successful or not). So what’s left? Starcraft. More specifically, its first expansion in years, so I’ve been covering my last two weekends not in photography, but in gaming.

Kind of rare in me, I have to admit; since the original Starcraft 2 I didn’t enjoy myself playing so much. Of course, what I’m doing right now is simply re-playing the whole campaign to be ready for the story and, as you might’ve heard, I’ve also isolated myself from watching the latest trailer because… well, quite simply, I remember a similar trailer when Wings Of Liberty launched that spoiled some scenes for me, and I really want to give this game a chance to enchant me, so I’m avoiding it.

That of course, didn’t stop me yesterday evening from asking myself: “is there any information about Starcraft II’s game engine?” It wasn’t exactly out of the blue; I was ogling this book for cues on C++ performance coding.

And there you go. Like magic. The video’s from 2011, so some work on HoTS was already underway by that point. Blizzard really is an elite playground for the most highly skilled game developers out there.

Makes you want to watch this, huh? :)

How NOT TO post-process a photo (HDR)

Tue, 19 Feb 2013 08:31:00 -0500

Okay, so we’re again back in the photography world, and I want to keep going. There’s still so much photography work waiting for me in the back-burner, but I feel this is a lesson I should share, too. As you can see above this post’s cover picture isn’t anything to talk at home about; it’s just plain boring. However, for me this building is nothing but boring: it’s the building that’s provided me a shelter in which to comfortably study and work time, and time again. And back in September (yes, September) I already knew I wanted to dedicate a special picture to it, so I decided one day I’d go early, park the car near the entrance (see the cars on the right?), and when the light got low enough, come out, keep my books in the boot and start taking pictures. How innocent of me.

I did manage to get some, in fact, but security wouldn’t let me, after I got close to the entrance, claiming I was taking pictures from within the building’s property (the small black patch of asphalt outside it). I’d taken a few, quite a few I’d say, and they weren’t allowing me to keep shooting because I needed a “special permit” to take pictures in their property, or so the officer said (as a note: he also said taking pictures inside is entirely prohibited because of IP issues with the building’s architects). Fine; I got the permit, and went down again to shoot even more pictures, but somehow I left feeling the “good pictures” where from my first shooting, not from my second.

Conversely, I also decided NOT to work on them after being finished; they needed some time, some experience; two things I wasn’t willing to give since I had more important priorities at the time. Until now, of course. Finally I decided I “wanted to make mistakes” with these pictures, and so I did.

This is our first winner for the evening. And believe it or not (easy to believe: the big streetlight’s shot of light is burned in the black ground) that’s an image straight out of the camera, no post-processing involved. It’s almost flawless: you’d want the light on the ground not to be burned down, and the library’s glowing light to be well-measured, too. Easy: I had a much lower-stop HDR shot in my Lightroom library, too. I merged both, did some “artwork”, and this was the result:

Yeah… no wonder none of my friends actually liked it on Facebook. I just went too far: I loved the detail HDR brought to the building, and the sky’s colors were a lot more intense, plus, the yellow haze from the streetlights was correctly reflected in the ground; I mean, that’s almost as real as it “could” get. But I went too far. Despite that, I just knew the angle and the composition were “killer”, so I put it up on Facebook anyway.

What follows though, is strictly between you and me.

Next up! After that killer shot, my number two shot reflected the main library’s beautiful entrance, showcasing the beautiful lighting inside and the particular entrance’s architectural style. I had many compositions at hand, all with multiple eVs to grant me all the post-processing freedom I could possibly need, and as I found out, all of my first-shooting’s were downright unusable. This is the first picture you’re seeing of my second shooting, and while the angles on my first were good, too, the streetlights posed a threat I didn’t want to deal with in Photoshop, to put simply (they were taken much later in the evening). Plus, I had a very strange sky filled with noise in my best frame and my lens wasn’t apparently clean enough, so I deemed those shots unrecoverable if I had something better, and I decided then that I had from my second field-day. Here’s the test HDR “build” I made to make that decision.

I agree with you, it’s beautiful, but if I could, I wanted to avoid myself all that Photoshop work (yes, I know it involves masking and erasing with multiple opacities to soften all of that stuff… but I just didn’t want to go there). You can decide later if my final picture is in fact better or not than this one; but that’s (in part) what this blog is all about: giving myself room to make mistakes and express them.

With my path chosen, I began work on that virgin shot I just showed you, and here is its first post-FX iteration:

Having arrived here, I’d already decided this was NOT a winner. I didn’t like how the (faked) sunset wrapped the building, and I wanted something better. With a little more work from the start perhaps, it might’ve been passable: after all, having made that previous (first) mistake, I knew I had to shoot for “realistic”, not “color-candyland”. But I decided to take my mistake even further, since the picture still didn’t provide enough of a “sunsetty” feel. So I took another pass and post-processed it again. Here’s the (eye-hurting) result.

Not what any human being would consider a pretty picture, right? I agree with you.

Since I didn’t entirely like the frame’s composition (I liked parts of it, like the fact it reflected the back of the entrance and the red arch behind, like the dusty HDR shot I showed you before), I moved on to another one, this one.

You can already tell it was a no-no, right? Oh, my….

Well, what I liked about this is that it reflected more of the entrance, of its character, and so it became my center of attention. First round?

The whole post-processing was more integrated with the picture, delivering better tones and colours. However, it was still a bit far on the candyland area: too much detail derived from the HDR process. And I didn’t like it. Small adjustments didn’t do it for me, either.

It was decent, ok? Maybe you disagree, but I think it was decent.

Still, that’s not what I wanted to portray of that building at all. It just wasn’t, no way. So I had to look further, and my next try turned out to be the winner.

But something had to change, because I was doing something wrong. I was still post-processing far too much, and I somehow had to keep the colors inline. My last attempt had proven to be acceptable, but had nothing of the beautiful and calmy feel that building gives me, and again, I wanted it to be a sunset, from the start. So we had work to do.

But boy, did it work out this time.

To achieve the sunset effect, I did not rely in any kind of filter, just common sense: I obtained an HDR image, and then set the highlights split tone to the appropriate color; only a few minor corrections were required, and the effect seems natural enough. The end-result is colourful, not the over-the-top variety of colors that makes you kill yourself, but in a beautiful kind of way instead. Four of my friends liked it on Facebook (five when I used it as my cover picture) and it has provided me with my second-highest rated picture in 500px. I still don’t think it’s without its flaws: namely, it would’ve looked much better without that vignetting, but somewhere in my post-FX chain I agreed to it and I couldn’t take it back once the end-result was obvious.

Lessons? I need to compare more often my pictures to the originals throughout the HDR stage (if any) and I need to keep the colors balanced, because this is my HDR style, not the over-the-top one. I have worse HDR-derived pictures (more painting-like) with three-to-four times more “likes”, and I’m starting to really grasp how “less is more” in photography.

I mean, HDR is necessary, at least it feels to me in landscapes, but it has to be subtle, only filling in those lights you’d miss if we weren’t talking HDR. Does this really feel HDR to you? No, after all you’ve seen. No. But it is.

With this post I want to explain the route I’m intending to follow after this lesson. Although, of course, I can imagine there might be a question lingering in your mind: “are you ready to finally fulfill what you once showed us?”

Definitely hope so. I’ll let you know the moment I get started.

Pyramid at (Fake) Sunset

Wed, 13 Feb 2013 11:17:00 -0500

I thought about working on this picture out of the blue, quite literally. It’s one of those pristine RAW shots I keep from the trip I took to SF for Google I/O 2011 (with the camera I can now call my own) from a time I knew next to almost nothing about photography. I took this one stopping right on my tracks during a zebra crossing and looking right to face this icon. It’s one of those pictures that has really stood out for me out of all the shots I took back then, and it suddenly hit me that I could do something about it.

The first thing I did though, was edit and photoshop the hell out of it, like I’d been doing during the first part of this year. The result is exactly what follows, a streamlined and flattered image of what could clearly be something a lot better. And I knew that, but I took the easy route to see what would come out on the other end.

The result isn’t dis-satisfying at all, isn’t it? It just looks too “coloured”. There’s nothing particularly wrong with it: it has its own look, and it steers your eyes towards something you’ve already seen, only highlighting very different colours this time.

I took a break, and thought about it some more. “I could do something really different with this picture!” I thought, and doing the exact same thing I’d been doing with already a bunch of pictures before wasn’t cutting it. Then I thought: “what’s the only thing wrong with this being ‘a perfect snapshot’?” The time of day: it’s bright afternoon, with a horrifying Sun hitting the pavement, instead of a creamy sunset. So sunset it was.

It took a while; I’m not going to lie. I researched (Googled) and looked out for help (YouTubed) until I found the method for turning day into night I liked the most: a “photography filter” on Photoshop (I’m on CS5.5), and I worked my way out from there. Colours weren’t always working for me, but I did my best to try and get the best look. The Sun was still giving the whole thing its glaring light, but it sort of reminded me of the last scene of Mamma Mia!, whom my Dad was quick to point out was shot on broad daylight using camera filters. The end result is here for you now.

The weakest parts of the image have got to be the pavement, with the harsh sunlight vying for attention, but it did turn out much better than I expected, and part of what I want to do with my photography is not only push for perfection, but to try different things, too. And since the image looked so keen for it, I also made my own “making-of” shot, comparing original and final side by side.

Looks even prettier this way, doesn’t it? But it sort of helps you visualize what I did and didn’t do to try to give the image as “sunsetty-look” as I could.

Since this (I posted it on my 500px on Jan 25) I’ve been hitting myself more traditionally with HDR, but in a different kind of way as you’ve seen lately.

More soon, I promise!

Google’s Notebook I know it’s quite easy to say...

Thu, 07 Feb 2013 17:55:00 -0500

Google’s Notebook

I know it’s quite easy to say “I’ve been thinking about this” when something not even remotely related to what you want to talk about spurs out of the vast ocean that is the web, but it’s true this time. And besides, I don’t need an excuse to talk about whatever I want to with you guys.

After watching countless Google I/O sessions, reading a ton of stuff on Google, tweeting with people I’ve never met IRL about the company and so on, one thing has always been clear to me: Google is fond of MacBooks. Now, from what I’ve heard, inside the company you’re free to work on either Mac or Windows, but one thing is very clear to me: Google adores Unix, and its favorite kind of Unix, is OS X.

Now, keeping this in mind, and remembering Google has been working on its own pieces of handheld technology since 2009 when they started building the Nexus One together with HTC, the question of whether Google would really love to build a notebook designed by and for its own engineers MUST arise. After all; Apple engineers have MacBooks, Windows engineers have Microsoft’s R&D Department building God knows what, and every other big tech company can basically use whatever they want. Facebook, I believe, with its hacker culture, must care little about which one you use, as long as you get things done. Start adding the prevalence of Nexus hardware being available (including the short-lived Nexus Q), and the thought of Google engineers making a notebook computer tuned for themselves catches some weight. But we’ve got a few issues to face first here.

As I mentioned, Google engineers seem to love MacBooks, and although Google’s products have lately demonstrated a tendency to not fall short of their goals, building a laptop with a touch & feel sensation you’re comfortable is only half of the work done. Slapping chips into the hardware isn’t a problem either; Intel would be happy to help you out in any way they can, out of fear of being blown out into history due to ARM’s increasing marketshare in the mobile computing space, and if you went the ARM route, of course, there’s plenty of powerful hardware there at your disposal. No; the problem is software.

Because obviously OS X cannot (and will not) be licensed to any other manufacturer, at least in the following months that’s not a trend anybody of us closely watching the Valley can bear the sight of. No. Is the answer Chrome OS, then? ARM hardware could run it, although I doubt Google engineers building their own hardcore laptop would want an ARM SoC instead of a powerful Intel/Nvidia-AMD combination. Or maybe, what’s important is not the performance, but what they want to use that laptop for. So. What does a top-notch Google engineer need?

Bare minimum? A working Unix console. If you have a Unix console, you can use Unix’s most powerful tools, write code on Vim, and commit using git. Whether you’d be able to actually debug what you’re coding is an entirely different matter, but that’s exactly what brings us to the point regarding this machine’s OS: if it’s Chrome OS, your coding is limited to the web, or maybe Chrome OS itself. The only viable (remaining) alternatives are either Linux or Windows, since OS X is out of bounds.

If this is indeed a Chromebook, I doubt it’s “made by and for” engineers, since if I were a Googler, I’d find it quite lacking as my main machine, unless I could get at least Windows 8 on it. About the touchscreen, it doesn’t really seem relevant to me, and if it really is 2560x1700, well, I bet it’s mainly a marketing scheme; nothing really to talk home about. I’ve said in the past I want a 1920x1080 “point” screen at 2x resolution, not 1440x900.

For now, these are the thoughts I’ve got on this latest assault from the web. Whether it proves to be real or not, is an entirely different subject. A “Pixel” as I/O 2013 gift? Those are going to be a few lucky attendants, really…

"An unforeseen consequence of ditching my smartphone for my brick-dumb Huawei is that I don’t have a..."

Wed, 06 Feb 2013 14:42:35 -0500

“An unforeseen consequence of ditching my smartphone for my brick-dumb Huawei is that I don’t have a camera on my phone. And so try to imagine this: I hung out with a girl for multiple months. Saw her multiple times a week. She even met some of my friends, I even met a few of her friends. And I don’t have a single photo of her.”

- So, I’ve been quite busy for the last 5 days with pictures. First it was with my own pictures, and then it was with a set of 300 pictures of my friends. It doesn’t look like it, but it is a lot of work for me; a lot of work that I incidentally love to do.

Whilst waiting for another assault of Facebook notifications, I started spinning my usual set of 10-35 Chrome tabs (yup) and one of then happened to be this article by “the man who’s (nearly) spent a full year off the Internet”.

Quite a story. Quite a paragraph; that must be the single most emotional piece of writing I’ve read in quite a while.

Formula One rolls out of the door You wouldn’t normally...

Sun, 03 Feb 2013 13:53:47 -0500

Formula One rolls out of the door

You wouldn’t normally expect me to be talking about Formula One, and much less, you’d expect to see me posting a video here about Red Bull Racing, the team that won both World Championships against Ferrari and Fernando Alonso just a few months ago. Yet, even though the Scuderia posted a video on Friday of their beautiful new car, the F138, I have to give it to the boys and girls of Milton Keynes, who definitely made a much better video.

For someone who lives to work in the world of motorsports, I know these videos are nothing else than “business”; in other words, the kind of things they have to do to actually pay the bills of what they really love to do: build cars that go really really fast, cars that abide by the rules (or are not caught if they don’t), and most importantly, cars which are faster and cleverer than the competition’s. That’s what really drives this kind of people.

Nevertheless, that doesn’t mean we just can’t sit and enjoy the show, right? Moreover, if Ferrari shot the pictures of the new F138 with a full suite of photographers armed with Canons and Nikons, Red Bull decided it wasn’t enough to just beat everyone else in the video department, but also in the cameras used for the promotion shots: medium-format Phase One cameras, people!!

Oh, I’m never going to have enough of this sport. When are you guys going to invite me to do some really good work for you? I bet I can make that new two-tone Ferrari colours shine better than anyone else ;)

Against The Informer(s)

Sat, 02 Feb 2013 12:53:05 -0500

Apparently, I’m not the only one who deeply misses Manhattan’s most flattering show in recent years. I thought I had it bad, chasing every week the ups and downs of a set of cool kids fresh out of high school, fresh out of college (wait! did they ever finish? I can’t remember a graduation episode…) who clashed with each other barely understanding what was the right way to cope with their feelings, but at the same time, I absolutely loved the show. It wasn’t entirely reasonable, the show barely made sense in the real world, but it made sense to me, because it got rid of anything else that wasn’t just “feelings” and it allowed its characters a “do what you want and get away with it” approach, no strings attached; it was all pure desire, emotion, and self-satisfaction. Something we barely get in a world where we’re more busy of being kind and taking care of others instead of pursuing what we really want and asking for it if it’s in someone else’s hands to give it to us: and yes, you’re allowed to take those words and give them the meaning you need them to have, that’s why I wrote for you.

Right, let’s cut to the chase.

About a week ago, I noticed my group of close friends started “liking” a page in Facebook, it was called “El Informer de la ULL”. Its cover picture and its name, translated to Spanish “The Universidad de La Laguna’s Informant” made me believe at first it was some kind of politics movement; an insider who was within the main campus of the University and who was happy to spill the beans on what exactly was going on with the money the University has been raking in, especially this year when they handsomely increased our tuition fees. Except it wasn’t that.

Instead, it was the mere apparition in my environment of a movement started at the Universidad Autónoma de Barcelona by which a group of people, or perhaps only one, decided they missed Gossip Girl too much, and turned it into a reality. But of course, this being 2013 (yes, it’s that fresh), opening up a blog and setting it up was too much work, so instead, a Facebook Page was the host of this new gossip hub. The way it works is the one you’d imagine: people send private messages to the [Facebook] page, which gathers all the messages, and posts them anonymously, more or less often depending on the amount of gossip gathered in the last few hours. The movement caught fire, and it wasn’t enough to have one per University, so there’s one per School; in fact, the image you see above is of the public Facebook Page of my own Computer Science School’s Gossip Facebook Page. Or “Informer” if you will.

I’m not going to lie. My first impression was sort of mixed. At first, my inner devil (yup; still not done with Fifty Shades Trilogy, and it shows) crawled out of my heart and began thinking about the kind of pranks you could play with this; after all, you could get away saying anything you wanted about anyone you wanted (hey! I just thought about making a joke to tease a few girl+friends [watch the ‘+’!!!], tell me you wouldn’t have thought something similar!), and then play “oh hey! I read this the other night on Informer… guess whom I think they’re referring to?”.

Then, it dawned on me: “I wish this page didn’t exist”. I wish none of the Informer pages existed, to be more precise.

Thing is, “with great power, comes great responsibility”, and this is so wrong on so many moral levels. Yes, it could be used to play pranks on the people you know & love in a new and exciting playground (for God’s sake, lock your computer’s screen EVERY TIME you leave it unattended, even if it’s just for a matter of seconds!!!), but at the same time, it wouldn’t be too comfortable for someone to get the impression he/she’s being “admired” (that’s the term I’ve been reading lately as a stand-in for ogling) now, would it? I mean, wouldn’t it make you uncomfortable to feel you have some kind of stalker constantly watching your moves and trying to get as much information about you as possible? (I know it can be considered “sweet” in special cases, but that’s not my point here.) Suddenly (and for example! don’t take me word for word!!), all of the things “hungry” (for lack of a better term) boys think upon seeing a pretty girl now have a very convenient megaphone to “inform” the whole campus; and this particular use of the “service” won’t just be confined to the classic gang of boys who look very tough when they’re together but can’t face an honest stare from a girl who has a little sense of self-respect for herself on a one-to-one. And it gets worse. Taken outside the “group of people who know each other”, or even the people who look at each other every day at School (you know, the classic “I know who you are but we’ve never really talked” look), but are mature enough not to take it any further, things can get amazingly out of control. Imagine this: you’re fresh out of high school and your hormones just don’t have any conscious deterrant against speaking out loud about a personal fallout with another freshman. Where do you think you’ll go to voice your, er, “concerns” regarding that other student? To your gang, or to the “Informer”? Worse still: at a high-school age, where it appears we humans are not designed to have any sense of empathy towards one another (not only boys on boys, girls can be pretty mean on other girls too), how do you think this kind of firepower will be used, and to what effects?

Consequently this opens two entirely new chapters in the book of how social networks are being misused, the first of which, is yet another dimension to the concept of bullying. You might’ve laughed at the whole concept of “Informer” pages being prominent in high-schools, but this is the kind of thing that just speaks to wannabe college students who want to get ahead of themselves and live up to the standards of the older kids, meaning kids who’re almost out of high school will use this kind of tool to their advantage, and leave behind a new kind of traumatised children. Because let’s be honest here; I’ve always been in favor of technology as a tool to connect us with our people, not to pin the dirt we may have on each other on a public board, which is basically what “Informers” are. And regarding high-school and early college, what do you think kids are most interested in? Sex, and most importantly, who has it and with whom. And if such news is already the kind of gossip you find out even if you try very hard to remain oblivious to, imagine if suddenly all of those bits of your past become letters written in an ink that spreads amongst hundreds of servers across the globe which will be there standing the test of time. No, this is not what I had in mind when I thought about how these “digital cutouts” could actually be good for our legacy. Not at all.

The second chapter is of course, the whole idea of “how much censoring are the so-called Informer(s) entitled to do?”. After all, if we were really against the idea of a certain individual (or set of) being responsible of all our secrets, we would choose another one and support that Informer by “liking” his/her page and sending him/her our messages to transmit, wouldn’t we? But of course, that’s half the point: we don’t really know who that Informer is, and if someone does, it could prove to be a very effective weapon. Who’s to say these Informers are not good-looking scams set to just gain power on campus and get sent all the dirt around town? No, people can’t be as textbook-characters as a teen drama portrays, right? Of course not! Because if there’s something I’ve learnt in twenty-five and-a-half years of existence, is that real life stories tear those of fiction when it comes to heartlessness.

Listen. I know this seems like a pretty big and grumpy rant, but I really, really believe, Gossip Girl shouldn’t exist as such in the real world. It worked on a TV show because it was only about a group of six kids and the people entering and exiting their lives; if it wouldn’t work as a show to have a gossip blog about a whole town, do you really think it’ll work with real people living in real towns and real cities? You think this is somehow going to improve your life in ANY level? No, it’s not. Don’t kid yourself. And if you don’t believe me when I imply people are, from what I’ve seen, “behaving” up to a certain point until now, I’ll use a real life example I got off one of the books I’m busy with: suppose you leave a five year-old car parked in a neighbourhood known for its disturbia, what do you think’ll happen after a week? Nope, not at all; the car will be untouched. But take a small rock and use it to break the car’s window; people will get the message “nobody cares about the car”, and they’ll start hammering it down, little by little. And by the end of the day you broke through the car’s window, its ashes will be the only thing left to remember that it really stood there.

Please, listen to me. Let Informers die, do not hand them over any more oxygen, and let’s not add these online local gossip boards to the ever-increasing list of disrespectful manners which already clutter our legacy. For our kids. Four our grandkids. For our future generations, please forget these sites exist. Live past them.

iPad 5 The thing I like the most about tumblr, is the wicked...

Tue, 29 Jan 2013 14:44:00 -0500

iPad 5

The thing I like the most about tumblr, is the wicked ways in which it allows you to present your own posts. This one for example, allows me to start by calling you out to look at the picture first, then read. If I do it otherwise, you’ll struggle to read first the post’s title, because before reading me, you instinctively want to check what this post’s about, and check if it’s worthy of your time. But here, I have the power about such simple things, haha. And I love it!

So, iPad 5. To begin with, I don’t necessarily give credit to these kind of “rumor posts” that begin the silly season every year talking first about an incoming iPad, and later in the summer about an incoming new iPhone. And its mini counterpart (talking about the second.)

Today though, I found this picture rather interesting: it made total sense. Dropping the “new iPad” in Spring of 2012 made total sense; updating it just a mere 6 months later with twice the power (in both fronts: CPU and GPU) didn’t; it made Apple look as if they were the ultimate performance junkie, when Steve proudly believed that PC makers trying to shoe their way into “mobile devices” were talking about speeds and feeds, which was the wrong way to get about these kinds of computers. (Yes, they are computers.) By the way, nothing to do with the fact Apple has been stumbling quite badly with Wall Street investors in recent months.

Continuing my derailment. Up until the iPhone 4, Apple released phones at WWDC. 2011 changed that with the 4S, but I suspected we’d see a new iPad come Spring 2012, and so we did. A new iPad, built on the same body as its predecessor and the venerable iPad 2, released months later didn’t make any sense at first, but then it did. So rather than a big mistake, it was actually a correction: the iPad 3 was the mistake being too early (too underpowered) at the party, but then Apple wanted to showcase their mobile business came before their Mac business, and so, the “Retina” moniker had to fall upon their full line of iDevices before it did to their premium line of Mac portables, because make no mistake, the classic MacBook Pros are going to die; I expect the “Air” to become just “MacBook” and the “Retina” models to drop the “Retina”. So sad there won’t be a 17” Retina MBP.

The consequences? Despite the rumor mill, I do not expect either a new iPhone nor a new iPad until the Autumn, where both lines were updated last year. Rumours pointed at a 128GB iPad; I never believed it’d happen so soon, but there you go: the 4th-gen iPad now comes in a 128 GB variant, and as of today, in fact. As for the new iPhone… could it switch gears and be announced at WWDC? I don’t discard it, though I don’t necessarily expect it. If it does, we’ll see the appropriate smoke, it’s like that these days. What I do expect is a little something called OS X 10.9, which, a year ago, was announced, to my shock, somewhere around this timeframe. So an announcement like that could hit us any day now, with 10.9’s release date & price being announced at WWDC, together with iOS 7.

Boy I hope iOS 7 overhauls the damn Operating System for once and for all; it’s full of UI-convention rot dating back to the original “iPhone OS” release of the original iPhone sold in June 2007.

So what’s in the cards for this new iPad? It finally marries together the design of both the iPhone, the iPad mini and the iPad itself. And by the looks of the picture, it seems its designed fell into our hands late last year, in the form of the cool new iPod Touch. For me, it means I finally have a good reason to shell out my money for the next iPad. Not that the Retina display wasn’t enough for me, given the ton of content I read with it and the importance I give to pixel density looking towards my eyes’ health, but it just didn’t feel right at the time. Then Apple came and threw the 3 out of its stores and supplanted it with an equally looking device although much stronger in every department, given its speed improvement permeates every experience with the device. Was it time then? No. Tempting, but no. It still keeps the 2nd-iteration design, and I knew a new design would come after the 3 (I didn’t entirely miss my mark; who expected two flagship iPads in a 6 month timespan!?), and this one does look the part of my iPad 2’s successor, the one I had to wait for outside of San Francisco’s Apple store a good 5 hours to get (I’ll keep a picture for another day. Promise!).

But there’s one thing I do not buy: the iPad Mini “as”, the better product. It’s not.

Here’s why: it’s the little brother.

From the moment I saw it through the livestreamed keynote I knew it would hook anyone who would make the mistake of laying their eyes upon it and holding one with their own hands, but I still kept repeating people the very same thing: if I had one, I’d know there’s a bigger one out there, and that wouldn’t satisfy me at all. Why? Because as it stands, the 9.7” iPad is just perfect for me. The only thing it doesn’t do well? Films, due to its 4:3 aspect ratio, but I’d rather have a 4:3 9.7” iPad than a 10.1” 16:9 (I flipped them to emphasize my point) Nexus 10, and I love the dual ARM Cortex A15 nature of the Samsung-built Google Nexus tablet, but widescreen isn’t for me in the tablet department. I can take it in a Nexus 7, but again, 7 inches is too small for me. The iPad 2 isn’t too heavy for me as some claim, it doesn’t tear its way into my flesh as some comments point out nor is it too big for me: I can carry it quite easily in my backpack, and I bought a case I can wear over my neck if I just want to walk with it as I did in Hong Kong. So what on Earth is the problem with it?

I use it to read, I appreciate its size, and it even gets small in some circumstances, but it’s good enough for most situations, and it surprisingly saves me from taking my 5kg 17” MBP if I want to do some light-study sessions where all I need are PowerPoint presentations (no more printing tons & tons of PDFs!!!!) and it keeps me company after hour of reading in my couch. It’s just perfect!!!

Now, I understand a smaller size has its advantages, but given what I do with it, for me, the big iPad is the winner. And the fact that the Mini is just a resized iPad 2 with an iPhone 5-esque body at the moment doesn’t make it better for me. It is a tablet I’d recommend though, make no mistake: I only recommend four tablets on the market, the Nexus’ and the iPads. The rest are plain rubbish from what I’ve seen, and I’m glad Google took matters on its own hands when it came to the horrible tablet experience marring the Android landscape.

So. The 5th generation iPad. Let’s wait and see how it develops throughout the year, and hopefully, I might be able to claim one of those as mine. Unlike some other things I said I might claim, but we’ll get back to that later.

Wait. I do that too much, don’t I?

Oh, my.