Happy Birthday to the First Program Run from Memory

Happy Birthday! The first computer program to be run from memory was 65 years ago today, June 21, 1948. The computer, the Manchester (UK) Small Scale Experimental Machine (aka, Baby) was the world’s first stored-program computer.

According to Wikipedia, It was 17 feet (5.2 m) in length, 7 feet 4 inches (2.24 m) tall, and weighed almost 1 long ton (1.0 t). The machine contained 550 valves—300 diodes and 250 pentodes—and had a power consumption of 3500 watts. It had a whopping 128 bytes of RAM. The machine was built by Frederic Williams, Tom Kilburn and Geoff Tootill.

Here are sample instructions:
LDN X // load negative X into the accumulator
SUB Y // subtract Y from the value in the accumulator
STO S // store the result at S
LDN S // load negative value at S into the accumulator

Click here for more info at Wikipedia.

ISS Switches to Linux

International Space Station Switches from Windows to Linux, for Improved Reliability

By  on May 9, 2013 at 9:21 am
original posting at extremetech.com

The United Space Alliance, which manages the computers aboard the International Space Station in association with NASA, has announced that the Windows XP computers aboard the ISS have been switched to Linux. “We migrated key functions from Windows to Linux because we needed an operating system that was stable and reliable.”

In specific, the “dozens of laptops” will make the change to Debian 6. These laptops will join many other systems aboard the ISS that already run various flavors of Linux, such as RedHat and Scientific Linux. As far as we know, after this transition, there won’t be a single computer aboard the ISS that runs Windows. Beyond stability and reliability, Keith Chuvala of the United Space Alliance says they wanted an operating system that “would give us in-house control. So if we needed to patch, adjust or adapt, we could.” It’s worth noting that the ISS laptops used to run Windows XP, and we know they’ve been infected by at least one virus in their lifetime: in 2008, a Russian cosmonaut brought a laptop aboard with the W32.Gammima.AG worm, which quickly spread to the other laptops on board. Switching to Linux will essentially immunize the ISS against future infections.

The laptops that were upgraded belong to the station’s OpsLAN. The crew use the OpsLAN to perform day-to-day activities, such as viewing stock inventory, controlling scientific experiments, or checking their current location. Presumably the laptops used to run bespoke Win32 apps on Windows XP, and now those apps have been re-written to work on Linux — hopefully they’re not being emulated in WINE. To get the astronauts and cosmonauts up to speed, they will be trained by the Linux Foundation.

To be honest, we shouldn’t be too surprised at the ditching of Windows. Linux is the scientific community’s operating system of choice. CERN’s Large Hadron Collider is controlled by Linux. NASA and SpaceX ground stations use Linux. DNA-sequencing lab technicians use Linux. Really, for applications that require absolute stability, which most scientific experiments are, Linux is the obvious choice. The fact that the entire OS is open source and can be easily customized for each experiment is obviously a very big draw, too.

Robonaut 2

In other news, the first humanoid robot in space, Robonaut 2, which also runs Linux, is due for an upgrade soon. Robonaut 2 (pictured above) was delivered on Space Shuttle Discovery’s final mission in 2011, and at the moment it’s just a torso with two arms — but later in 2013, some climbing legs and a battery pack should be delivered. The ultimate goal is to see whether humans and robots can operate peacefully in zero gravity, with Robonaut eventually performing menial tasks (vacuuming, changing filters), and possibly dangerous tasks during space walks, too.

Out- sourcing Your Own Job

Pretty funny but damn, this takes balls, and a complete lack of conscience or regard for co-workers. A developer at Verizon was busted for outsourcing his entire job to China (he FedEx’d his VPN key fob to a Chinese company who then used it to log in as him). While someone in China was writing his code, he sat at his desk and surfed all day.

You can read the story at Huffington Post. I have copied and pasted it below in the event the link dies. Below that is the actual case study from Verizon (from which HuffPost, et al, gathered their material for their write-ups). They wrote it up as why it might be good to practice pro-active log reviews (the employee was caught due to logs showing VPN logins coming in from China which were subsequently thought to be hackers).

Unfortunately, this fellow probably did a HUGE disservice to his fellow employees. He has alerted management and now Verizon is probably thinking, “Wait! We could outsource this entire department.” Hell, they are probably already in negotiations with the Chinese company. While at face level his idea may seem clever, as soon as you scratch the surface, you realize this guy deserves a Douchebag of the Year award.

========================================
Developer Outsourced Entire Job To China, Spent Hours Surfing The Web, Watching Cat Videos
The Huffington Post | By Meredith Bennett-Smith
Posted: 01/16/2013 4:31 pm EST | Updated: 01/16/2013 5:14 pm EST

A crafty developer reportedly figured how to get paid to sit and watch cat videos for a good chunk of the day.

It’s a story almost too good to be true — and one which has an almost uncanny resemblance to this fake news story run by The Onion. But according to Verizon’s Security Blog, a U.S. developer actually did find a way to fool everyone at his company into thinking he was working, while in fact outsourcing his entire job to China.

Andrew Valentine wrote up the case study for Verizon, and the story apparently caused such a furor it temporarily crashed the Verizon servers.

Citing the study, the BBC notes the ingenious scam came to light after the employee’s company asked for an audit to investigate “anomalous activity on its virtual private network (VPN) logs” that pointed to an active VPN connection between Shenyang, China, and the employee’s workstation that appeared to be operational for months.

Valentine went so far as to profile the employee, who is not named in the report, and who was paying less than “one fifth of his six-figure salary” on the outsourcing:

Mid-40’s software developer versed in C, C++, perl, java, Ruby, php, python, etc. Relatively long tenure with the company, family man, inoffensive and quiet. Someone you wouldn’t look at twice in an elevator.
A check of the employee’s web browsing history revealed an average schedule. According to the case study, the worker’s day looked like this:

9:00 a.m. – Arrive and surf Reddit for a couple of hours. Watch cat videos
11:30 a.m. – Take lunch

1:00 p.m. – Ebay time.

2:00–ish p.m – Facebook updates – LinkedIn

4:30 p.m. – End of day update e-mail to management.

5:00 p.m. – Go home

According to The Register, the employee no longer works for the company that ordered the audit. (As Gizmodo’s Jamie Condliffe quipped, “Looks like he’ll be spending more time on LinkedIn from now on.”)

Help Net Security reached out to Nick Cavalancia, a vice president at SpectorSoft, to gather information on how companies may work to prevent similar schemes.

“We have yet to see what impact this incident will have, but providing programming code used to run critical national infrastructure providers’ systems to off-shore firms seems dangerous at best,” Cavalancia said. “What many organizations fail to understand is that with proactive monitoring that can alert IT security teams when unacceptable online behaviors occur, this type activity can be thwarted before it becomes an incident.”

==============================================================================
Case Study: Pro-active Log Review Might Be A Good Idea
http://securityblog.verizonbusiness.com/2013/01/14/case-study-pro-active-log-review-might-be-a-good-idea/

Andrew Valentine
January 14th, 2013
With the New Year having arrived, it’s difficult not to reflect back on last year’s caseload. While the large-scale data breaches make the headlines and are widely discussed among security professionals, often the small and unknown cases are the ones that are remembered as being the most interesting from the investigators point of view. Every now and again a case comes along that, albeit small, still involves some unique attack vector – some clever and creative way that an attacker victimized an organization. It’s the unique one-offs, the ones that are different that often become the most memorable and most talked about amongst the investigators.

Such a case came about in 2012. The scenario was as follows. We received a request from a US-based company asking for our help in understanding some anomalous activity that they were witnessing in their VPN logs. This organization had been slowly moving toward a more telecommuting oriented workforce, and they had therefore started to allow their developers to work from home on certain days. In order to accomplish this, they’d set up a fairly standard VPN concentrator approximately two years prior to our receiving their call. In early May 2012, after reading the 2012 DBIR, their IT security department decided that they should start actively monitoring logs being generated at the VPN concentrator. (As illustrated within our DBIR statistics, continual and pro-active log review happens basically never – only about 8% of breaches in 2011 were discovered by internal log review). So, they began scrutinizing daily VPN connections into their environment. What they found startled and surprised them: an open and active VPN connection from Shenyang, China! As in, this connection was LIVE when they discovered it.

Besides the obvious, this discovery greatly unnerved security personnel for three main reasons:

They’re a U.S. critical infrastructure company, and it was an unauthorized VPN connection from CHINA. The implications were severe and could not be overstated.
The company implemented two-factor authentication for these VPN connection. The second factor being a rotating token RSA key fob. If this security mechanism had been negotiated by an attacker, again, the implications were alarming.
The developer whose credentials were being used was sitting at his desk in the office.
Plainly stated, the VPN logs showed him logged in from China, yet the employee is right there, sitting at his desk, staring into his monitor. Shortly after making this discovery, they contacted our group for assistance. Based on what information they had obtained, the company initially suspected some kind of unknown malware that was able route traffic from a trusted internal connection to China, and then back. This was the only way they could intellectually resolve the authentication issue. What other explanation could there be?

Our investigators spent the initial hours with the victim working to facilitate a thorough understanding of their network topology, segmentation, authentication, log collection and correlation and so on. One red flag that was immediately apparent to investigators was that this odd VPN connection from Shenyang was not new by any means. Unfortunately, available VPN logs only went back 6 months, but they showed almost daily connections from Shenyang, and occasionally these connections spanned the entire workday. In other words, not only were the intruders in the company’s environment on a frequent basis, but such had been the case for some time.

Central to the investigation was the employee himself, the person whose credentials had been used to initiate and maintain a VPN connection from China.

Employee profile –mid-40’s software developer versed in C, C++, perl, java, Ruby, php, python, etc. Relatively long tenure with the company, family man, inoffensive and quiet. Someone you wouldn’t look at twice in an elevator. For the sake of case study, let’s call him “Bob.”

The company’s IT personnel were sure that the issue had to do with some kind of zero day malware that was able to initiate VPN connections from Bob’s desktop workstation via external proxy and then route that VPN traffic to China, only to be routed back to their concentrator. Yes, it is a bit of a convoluted theory, and like most convoluted theories, an incorrect one.

As just a very basic investigative measure, once investigators acquired a forensic image of Bob’s desktop workstation, we worked to carve as many recoverable files out of unallocated disk space as possible. This would help to identify whether there had been malicious software on the system that may have been deleted. It would also serve to illustrate Bob’s work habits and potentially reveal anything he inadvertently downloaded onto his system. What we found surprised us – hundreds of .pdf invoices from a third party contractor/developer in (you guessed it) Shenyang, China.

As it turns out, Bob had simply outsourced his own job to a Chinese consulting firm. Bob spent less that one fifth of his six-figure salary for a Chinese firm to do his job for him. Authentication was no problem, he physically FedExed his RSA token to China so that the third-party contractor could log-in under his credentials during the workday. It would appear that he was working an average 9 to 5 work day. Investigators checked his web browsing history, and that told the whole story.

A typical ‘work day’ for Bob looked like this:

9:00 a.m. – Arrive and surf Reddit for a couple of hours. Watch cat videos

11:30 a.m. – Take lunch

1:00 p.m. – Ebay time.

2:00 – ish p.m Facebook updates – LinkedIn

4:30 p.m. – End of day update e-mail to management.

5:00 p.m. – Go home

Evidence even suggested he had the same scam going across multiple companies in the area. All told, it looked like he earned several hundred thousand dollars a year, and only had to pay the Chinese consulting firm about fifty grand annually. The best part? Investigators had the opportunity to read through his performance reviews while working alongside HR. For the last several years in a row he received excellent remarks. His code was clean, well written, and submitted in a timely fashion. Quarter after quarter, his performance review noted him as the best developer in the building.

This entry was posted on Monday, January 14th, 2013 at 2:46 pm and is filed under Editorial. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.

U.S. Dept of Home- land Security & Java

Wow. Java and Oracle really getting beat up lately. Java seems to be following the downward spiral of Adobe Flash (in terms of viability in being used for web apps/websites). First, Apple sticks a fork in them, and now the government. Homeland Security has recommended that users disable Java in their web browsers. Ouch. That hurts.

The full story at NBC News. I’ve copied and pasted it below in case the link dies.

=======================================================

US warns on Java software as security concerns escalate
By Jim Finkle

The U.S. Department of Homeland Security urged computer users to disable Oracle Corp’s Java software, amplifying security experts’ prior warnings to hundreds of millions of consumers and businesses that use it to surf the Web.

Hackers have figured out how to exploit Java to install malicious software enabling them to commit crimes ranging from identity theft to making an infected computer part of an ad-hoc network of computers that can be used to attack websites.

“We are currently unaware of a practical solution to this problem,” the Department of Homeland Security’s Computer Emergency Readiness Team said in a posting on its website late on Thursday.

“This and previous Java vulnerabilities have been widely targeted by attackers, and new Java vulnerabilities are likely to be discovered,” the agency said. “To defend against this and future Java vulnerabilities, disable Java in Web browsers.”

CERT’s instructions on how to do so can be found here, under “Solution.”

Oracle declined on Friday to comment on the warning.

Paul Wagenseil, senior editor for security at TechNewsDaily, writes that “it’s easier than ever before to disable Java in browsers. The latest version of the Java Control Panel for Windows has a checkbox under the Security tab labeled ‘Enable Java content in the browser.’ Uncheck that and all your browsers should be Java-free.”

If you are running an earlier version of Java 7 for Windows, “you’ll have to disable each browser individually.”

To make sure Java is really disabled, Wagenseil writes, users can visit this site to check.

“For versions of Java older than Java 7 (which you shouldn’t be running anyway), the de-Javafication process for Internet Explorer involves editing the Windows Registry,” he notes. “If you don’t know what that is, don’t do it. Instead, stop using Internet Explorer entirely.”

Wagenseil says that “Unless you use Java professionally — such as by developing Web or Android apps, updating a Website or using Adobe’s Creative Suite software package — you don’t really need it.”

What is Java?

Java is a computer language that enables programmers to write software utilizing just one set of code that will run on virtually any type of computer, including ones that use Microsoft’s Windows, Apple’s OS X and Linux, an operating system widely employed by corporations.

Computer users access Java programs through modules, or plug-ins, that run Java software on top of browsers such as Internet Explorer and Firefox.

The U.S. government’s warning on Java came after security experts warned on Thursday of the newly discovered flaw.

It is relatively rare for government agencies to advise computer users to completely disable software due to a security bug, particularly in the case of widely used programs such as Java. They typically recommend taking steps to mitigate the risk of attack while manufacturers prepare an update, or hold off on publicizing the problem until an update is prepared.

In September, the German government advised the public to temporarily stop using Microsoft’s Internet Explorer browser to give it time to patch a security vulnerability that opened it to attacks.

Prime target for hackers

Java is so widely used that the software has become a prime target for hackers. Last year Oracle’s Java surpassed Adobe’s Reader software as the most frequently attacked piece of software, according to security software maker Kaspersky Lab.

Java was responsible for 50 percent of all cyber attacks last year in which hackers broke into computers by exploiting software bugs, according Kaspersky. That was followed by Adobe Reader, which was involved in 28 percent of all incidents. Microsoft Windows and Internet Explorer were involved in about 3 percent of incidents, according to the survey.

The Department of Homeland Security said attackers could trick targets into visiting malicious websites that would infect their PCs with software capable of exploiting the bug in Java.

It said an attacker could also infect a legitimate website by uploading malicious software that would infect machines of computer users who trust that site because they have previously visited it without experiencing any problems.

They said developers of several popular tools, known as exploit kits, which criminal hackers use to attack PCs, have added software that allows hackers to exploit the newly discovered bug in Java to attack computers.

Similar scare last August

Security experts have been scrutinizing the safety of Java since a similar security scare in August, which prompted some of them to advise using the software only on an as-needed basis.

At the time they advised businesses to allow their workers to use Java browser plug-ins only when prompted for permission by trusted programs such as GoToMeeting, a Web-based collaboration tool from Citrix Systems.

Java suffered another setback in October when Apple began removing old versions of the software from Internet browsers of Mac computers when its customers installed new versions of its OS X operating system. Apple did not provide a reason for the change and both companies declined to comment at the time.

Adam Gowdiak, a researcher with Polish security firm Security Explorations, told Reuters he believes that Oracle fails to properly test its software fixes for security flaws. “It’s definitely safer for users to stay away from Java ’til Oracle starts taking security seriously,” he said.

(Editing by Dan Grebler)

(c) Copyright Thomson Reuters 2013.

NYT Magazine Web Article – “Snow Fall”

Check out Snow Fall, a New York Times Magazine online article. Great use of HTML5/CSS3/jQuery, and a nice, clean, simple design.

Make sure to interact with the photos. And there are several chapters to view (accessible via the top nav and at end of each story).

I love how they are translating the magazine app feel onto a website. We’re going to see a lot more of this. Much easier than apps – for both consumer and producer.

And better yet, here is the case study – a Q&A with the New York Times team themselves.

Rudy

/bin/rm -r -f * at Pixar

Here is a great story that I had not heard until now. Back in 1998, while working on Toy Story 2, Pixar accidentally deleted a huge chunk of the files with the Unix command rm -r-f *. On top of that, their backup solution failed. Oopsie.

The whole story, “How Pixar’s Toy Story 2 was deleted twice, once by technology and again for its own good”, is over at The Next Web. Written by Matthew Panzarino, it’s a really great geek read. It is also a great study in how best to handle a crisis situation. Good culture, good company.

And there is a Pixar animation about the whole affair over on YouTube.

I’ve included the whole story below in the event the link vanishes…
====================================================================================

By Matthew Panzarino on 21 May ’12
How Pixar’s Toy Story 2 was deleted twice, once by technology and again for its own good

“That’s when we first noticed it, with Woody.”
“[Larry Cutler] was in that directory and happened to be talking about installing a fix to Woody or Woody’s hat. He looked at the directory and it had like 40 files, and he looked again and it had four files.”
“Then we saw sequences start to vanish as well and we were like, “Oh my god”
“I grabbed the phone…unplug the machine!””
That’s Oren Jacob, former Chief Technical Officer of Pixar—then an associate technical director for Toy Story 2—recounting the moment they discovered that the movie was being deleted off of the company’s servers after an erroneous command was executed, erasing two months and hundreds of man-hours worth of work.
You might have heard something about this lately, as a clip from the special features of the movie has been making the rounds after being posted on Tested. It’s narrated by Jacob himself, and the movie’s Supervising Technical Director Galyn Susman.

The story struck me as interesting, so I reached out to Jacob, who is now the CEO of ToyTalk, a digital entertainment startup that is in the process of readying its first project for launch. I wanted to get the story right from the horse’s mouth, to see if the situation was really as dramatic as it had sounded, how the staff coped and whether or not they ever discovered exactly who deleted the files in the first place. As you can hear in the video, Jacob has a hyperkinetic conversational patois and, despite what he says, a great memory for the details of the situation.
A huge chunk of Toy Story 2 was indeed deleted and was only recovered by a stroke of luck and the intense efforts of the Pixar staff.
But what most people don’t know is that the whole movie was actually tossed out again, not by the computers, but by the filmmakers themselves. It was then completely remade with mere months to go before a release date that was set in stone, cementing Pixar’s legacy as a crucible of commitment to quality.
The story that Jacob shared with me ended up containing some interesting lessons for people working with large amounts of technical data, but more than that, it has a lot to say about just how much of what makes Pixar’s movies so great has to do with the people who work there and their insane dedication to making things great.
/bin/rm -r -f *
The story likely takes place in 1998, though Jacob admits he’s foggy on the exact date. The Toy Story 2 crew, about 150 people in the animation, lighting and modeling departments of Pixar, had been hard at work for some time on the movie. Simultaneously another 200-250 people were at work finishing up Bug’s Life, which would be released that Fall.
One day, Jacob (pictured right) was in the office of Larry Cutler—along with Larry Aupperle, who was also an associate Technical Director working under Susman. In what is a crazy stroke of luck, they happened to be looking at a directory in which the assets for the character Woody were stored, when they noticed, on a refresh, that there were suddenly less and less files.
“He had an error, I forget the exact [one]. It was like, “Directory no longer valid,” because he’s in a place that had just been deleted. Then he thought to walk up [a directory] and he walked back up and then we saw Hamm, Potato Head and Rex. Then we looked at it again and there was just Hamm and then nothing.”
The command that had been run was most likely ‘rm -r -f *’, which—roughly speaking—commands the system to begin removing every file below the current directory. This is commonly used to clear out a subset of unwanted files. Unfortunately, someone on the system had run the command at the root level of the Toy Story 2 project and the system was recursively tracking down through the file structure and deleting its way out like a worm eating its way out from the core of an apple.
That’s when the panicked call was made to the machine room where the main server was located and the instruction given to just yank the power and network connection of the server. This is simply not done in environments with hundreds of clients connected to the machine, it’s as if someone asked you to throw your main breaker to shut off your blender.
“The master machine goes down,” says Jacob. “Some people are animating a shot and they can work for like a minute or five minutes, but eventually you’ll have to pull data from the master machine for some reason or another, which your machine will freeze.”
“Eventually every animator and , every TD, everyone working on the show goes, “Oh, all machines down. Lets go to lunch. Ha, ha.”
The machine was eventually brought up a few hours later and they took a poll of the damage. When a size command was run on the Toy Story 2 directory, it was only 10% of the size it should have been.
90% of the movie had been deleted by the stray command.
“The show’s been trashed”
When this story originally started making the rounds, one of the other big questions was ‘how did this happen?’.
I asked Jacob about the ‘how’ and he told me that it was actually largely a function of how a company like Pixar works on projects.
“You have 400 people on the network and they all have to have like pretty massive access across the board to the whole project, so it’s hard to like, limit the damage,” Jacob said. “It could happen from almost any terminal.”
“Pixar being a wide open Unix environment meant that it was very promiscuous. You could [change directory] ‘slash’, net ‘slash’ or walk across the network and log into Ed Catmull’s machine or Steve Job’s machine if you wanted to. Not that Steve ever did do any work on the film directly, but you could do that.”
The common way to prevent an accidental command like this being run on an entire project is to lock users down with permissions to only the files they need. But, because of the way a project like a Pixar film works, almost everyone working on the show needed permissions to read and write to the master machine. This was their job.
Assigning micro-managed permissions would have eaten up administrative resources, especially in crunch time.
Backup plan
So at this point, most of the film had been deleted or otherwise compromised. But that wasn’t a big deal. Things had been deleted before, it’s just something that would happen from time to time. During the production of A Bug’s Life, most of the ants got deleted and had to be restored, which wasn’t a problem because, of course, Pixar backs up its data.
In 1998, the most common way to back up a bunch of data was on tape, which is the system that Pixar was using. Unfortunately, these backups were not continuously tested, as the company does today and is the universally recommended best practice.
Typically, to make sure your backups are good, you have to use them. Every few days or weeks, you swap your backups with your currently running setup and keep going, in order to make sure that your data is all there. This is a practice called ‘live backups’.
Pixar, at the time, did not continuously test their backups. This is where the trouble started, because the backups were stored on a tape drive and as the files hit 4 gigabytes in size, the maximum size of the file was met. The error log, which would have told the system administrators about the full drive, was also located on the full volume, and was zero bytes in size.
This meant that new data was being written to the drive, but it was ‘pushing’ the older files off. But no-one at Pixar knew this yet.
It’s worth mentioning at this point, because some of you may be wondering, that the whole movie encompassed no more than around 10 gigabytes of information. That may seem crazy considering the size of textures for many newer flicks, but you have to remember that the backup tape had a file-size limit of 4GB and it didn’t become a problem for many months on the project. The entirety of the data for the movie could have been fit on a couple of dual-layer DVDs.
So, they grabbed the backups, went to work and restored the show. Within a couple of days they had what they thought was a completely restored version of the files for TS:2.
To test it, they submitted around 2,000 frames to render, one for each ‘shot’ in the movie (the bits between ‘cuts’). This would effectively pull on every resource involved in the film because those stills would need all of the models, lighting and textures in order to render properly.
Everything looked fine. “We lost a week of a work,” Jacob says. “So those last 10 shots are the last week, but other than that…O.K..”
Fast-forward to the end of that week. The crew has been back at work using the newly restored files for many days now. But, over the course of that week, there had been a few oddities. Weird ‘attach’ errors kept cropping up.

An attach is when a character, like Woody for instance, takes off his hat. The hat transfers from being a part of his head to being a part of his hand, this is a tricky procedure and very ‘fragile’.
“We started doing comparisons of the shots and realized that the show was incomplete. How it had worked for that week and how such renders came out, I can’t explain.”
By the end of that week enough things had broken here and there that the team realized there was a problem. In addition to the attaches, some people working on a version of their shot noticed that the current version was far lower than where they had left off. They were working on number 420 and now it said it was version 20. Something was up.
This is when the tape backup issue was discovered, after a full week’s work.
“That work is definitely wasted, because it’s on top of an unreliable restoral,” recalls Jacob. “Now sadly, what’s happened is that there is zero confidence in any solution, because the restoral is bad, the work on it is bad, the deletion was horrible, and the backup tapes are busted.”
“All possible directions to move are broken and, maybe worse. We don’t quite understand how they’re broken. If only 10 percent of the show is not on the tape, which 10 percent? I don’t know.”
“That was the big meeting, in the conference room back in Bugville (Pixar’s corporate complex). All the big brains in the studio are like, “Uh, I don’t know. Oh my God!”
That’s when Susman said, “I have a machine back at my house.”
The $100,000,000 Volvo
Susman, the Supervising Technical Director on Toy Story 2 (pictured below), had given birth to her son Eli shortly before, and had been working from home. This meant that she had a Silicon Graphics workstation at her house. It was either an Indigo 2 or an Octane, pictured right, and it was loaded up with a full copy of the movie.
In order for her to work on the movie while out, they had plugged the machine up to the local network and copied the whole file tree over. Then she would receive incremental updates over her ISDN internet connection. For those not in the know, that was like two 56kbps modems duct taped together (welcome to 1998).
The last update that her machine had gotten could have been as old as a couple of weeks, but at this point the Pixar team had an incomplete backup and a corrupt tree full of files, and they needed anything they could get their hands on to fix the problem. This was the difference between rebuilding every missing file from scratch and, well, shipping the movie on time.
So Jacob and Susman hopped into her Volvo and shot back over the bridge from Richmond to her house to retrieve the computer. They hauled it out to the car and carefully placed it in the back seat, wrapping it in blankets and strapping it in tightly with seatbelts.
“There was nothing else to do,” Jacob says of the session described earlier. “We were dead. We’d been in the meeting for like 45 minutes. There was 30 of us, all the biggest brains Pixar can bring to the problem.”
That’s when Susman remembered her machine at home.
“She and I just stood up and walked out, back to her Volvo, drove across the bridge, got the machine, got some blankets, I hugged it with seatbelts, across the back seat. Drove at like 35 with blinking lights on, hoping to get a police escort. No cops saw us, so it didn’t help us.”
At that point, the Volvo had become a $100M machine, as the entirety of the team’s efforts so far on the project were ensconced on its drives.
They made it back to Richmond in safety. “Eight people met us with a plywood sheet out in the parking lot and, like a sedan carrying the Pharaoh, walked it into the machine room.”
They sweated as the machine booted up, as that’s exactly when most drives crash. It booted. They didn’t pass go, they just plugged it into the network and copied the entire drive off immediately, then starting picking apart what they had.
The backup was about two weeks old, but they were able to make it the ‘B’ tree to compare with the ‘A’ backup from two months ago and a third ‘C’ source that was cobbled together out of any local backups animators or modelers had made on their personal terminals, collected by ‘groveling’ for .old, .sav, .bak and any other old file they could find.
They managed to verify about 70,000 files, leaving 30,000 files to check, and they had to be done by hand. “We worked from Friday till Monday morning, nonstop through, in rotating shifts with food and sleeping bags, with about 10 or 12 of us,” recalls Jacob.
“When people came in on Friday, when they showed up. We’d hand them a printout of, “Here are the 500 things you’re checking in the next eight hours. Start running xdiff commands. Go for it.”
“Quickly, within a couple of hours, the scripting nerds had put together scripts that would ingest the list and spawn off XF windows, 20 deep. Close them all, 20 deep. Close them all, so you could look at them really fast.”
They all had to be looked at with human eyes, to see which ones were shorter or more current. They did it over the next few days. The feeling of sympathy and support are what Jacob recalls most vividly. Not just that employees had to sacrifice a weekend with their family and come in on the weekends, staying up late hours and even sleeping on site, but the sense of ‘digging in’ to fix the problem.
“We dug so deep at that point in time. People who were on the “Toy Story” crew, people on “A Bug’s Life,” and the folks in the studio at large. The whole community, whether that was supporting people who were working late, or on the keyboards who typing, or folks sending us food by messenger pigeon.”
“At some point, even some of the neighboring small little sandwich shops in Point Richmond  said, “You need free food today? I know you guys aren’t sleeping right now.”
The insane amount of focus needed to pull off comparing those files shows just how deep the dozen or so people on the project had to dig. The experience transcended employment and moved into the realm of true dedication to the movie, to their digital friends and to each other.
“The parts of the weekend that I can remember specifically are the trays of cookies, the lemonade, the pizza, and flowers that were sent,” Jacob recalls. “Somebody hired a masseuse on a Sunday to walk around for a while. Some other person happened to work at an emergency shelter and brought in blankets.”
They then rebuilt and tested the project and it ended up working, sort of. To this day, Jacob can’t explain the fact that more than several thousand files were missing from the tree by the time they were done.
“Where the files went, we don’t know. The fact that it still worked without them is totally unexplainable.”
But the project worked, the frames rendered, Toy Story 2 lived again.
Witchhunt
One of the big questions that I wanted to pose was whether or not it was ever discovered who was responsible, and whether they were punished. Normally when something like this happens, there’s a cry for accountability. Item one on the agenda is typically ‘who do we blame?’ Not so at Pixar.
“There was no attempt to hide it,” says Jacob. “We sent email out like 10 minutes later to everyone in the building. “Help. Holy s**t!”
Aside from the fact that there was immediate chatter about  who might have made such a dumb move, the discussion quickly moved right on to how to fix the problem.
“Let’s put the witch hunt away. We’ve got to get the show back first. Let’s not go spend a week of our time trying to kill somebody. Where’s the movie?”
“Obviously, five minutes in the meeting, you’re all sweating and red-faced. And somebody will say, “Let’s go kill somebody and lynch them. Now,” says Jacob, “I support lynching on our agenda. But, number one is, just get the movie back and work on Buzz and Woody again.We’ve lost our friends.”
With this many man-years, or even man-decades, worth of work on a project, the temptation to find someone to blame, to expend effort on hunting down the person responsible, is intense.
But that kind of negative thought process doesn’t help anyone and it just removes focus from what matters most: moving forward.
The systems administrators definitely “went through some deep soul-searching” about the backup plans and came to the big production meeting with a new backup plan in place, which was talked over very thoroughly. But there weren’t any summary firings or screaming matches.
Jacob can’t recall who on the executive staff was on staff the day that the backups were being restored, but he says that whoever was there, Steve Jobs, founder Ed Catmull and the rest of the executive staff were very supportive of the restoral efforts, rather than focusing on slashing and burning staff over the error. They bought the team Pizza that weekend, got them anything they wanted and were generally supportive.
During the big meeting over the backup problem Catmull, who is known for leading an ‘incredibly calm and zen-like existence’, simply asked what the team was doing about the issue.
Jacob recalls the exchange:
“Ed, we’re doing everything we can right now. OK?,”
“You guys keep on that problem?
“OK, thank you, Ed.”
The thing about  a disaster like this one is that the technical directors and staff at Pixar had to trust one another to fix the issue, even though there were several mistakes made and one of them was responsible. “If you can’t sit down and calmly engage that meeting, you can’t be in that meeting with them,” says Jacob. ”Because the circumstances were so incredibly unusual. Black Swan events do occur.”
Instead of dwelling on pinning the blame or lamenting the loss of time and effort, the team made sure to alter the backup strategy so that something like that didn’t happen again, and it went about making up for lost time.

Toy Story 2 gets trashed again
After the deletion and restoration of Toy Story 2, the team was likely hoping for an uneventful path to release, but it was not to be.
In the Christmas of ’98, after the release of A Bug’s Life and the promotional tour was done, John Lasseter, Andrew Stanton, Pete Docter and legendary story man Joe Ranft all came to the production team to take a look at Toy Story 2.
It was not a good film. They dedicated the winter vacation to re-writing the project almost entirely from the ground up. Production shut down on December 15th and came back after New Year’s in January, when the story team re-pitched the movie.
Lasseter and Lee Unkrich ended up co-directing the film along with Ash Brannon as it was seen in the theaters.
Among the things that stayed? The main characters, of course. Buzz, Woody, Hamm, Potato Head, Rexx. Andy’s room stayed. The Al’s Toy Barn sequence stayed. That’s it, nearly everything else you see in the film as it is new.

Jacob explains what was added, including the entire character and animation for Buster the dog:
Effectively all animation was tossed. Effectively, all layout was tossed. So all camera work would start from scratch. Lighting was in the film a little bit, but that was tossed as well. We had to build new characters.
So at that point, Buster showed up at that point. And that character went from being out to being in the screenplay to in the final screen in nine months.
That’s a fully animated quadriped…On the fly. And most of the humans in the film and show. All the background extras in the airport at the end”
They were all built and assembled then. And all the effects work was added to the film. The opening of the film, which is Buzz, Buzz playing with the robots, which I spent a lot of my time working on, where Buzz blows up a quarter-million robots with that crystal…that explosion. That was all added in that pitch as well. It started from ground zero in January.
So the story, effectively. And the film. And that was probably one of the biggest tests of what Pixar was as a company and a culture we ever went through.
The big deal about re-building the movie? It had a hard-set release date of November 22, 1999. That date was set in stone. A big-budget movie like Toy Story 2 has countless marketing tie-ins, promotional efforts and more that had to be timed perfectly with the release of the movie.
Moving the release date of the movie within a year is insanely difficult. Moving it within 6 months is impossible. This meant that the team had to re-make Toy Story 2 in 9 months. All because they wanted to make the best thing they could possibly make.
Disney didn’t believe that they could do it. But they did it anyway.
“At that point, you’ve still got to go, “How far are we going to take this?” Right?,” Jacob adds. “I mean, that’s balls to the wall.”
“January to September ’99 was an unbelievable Herculean effort to pull that show off from ground up again. That was one of the defining cornerstones to what Pixar as a corporate culture meant to itself inside. And what it could produce was that. Just because of that.”
Pulling a hundred-hour week now and then is tough enough on morale and physical well-being. But to do it for 9 months in a row, that was beyond the call of duty.
Pixar was also an independent and publicly traded company at that point in time. To blow a film like Toy Story 2, not release it on time, would have decimated the studio’s credibility, and detonated the Disney movie economy.
“Save Buzz and Woody. Save the franchise. Save the movie. Save the company. It was an all-in bet.”
Toy Story 2 was indeed finished and released on time. It grossed nearly $500M worldwide, was nominated for an Academy Award and cemented Pixar’s reputation as the studio that wouldn’t compromise.
Lessons learned
In closing, Jacob told me that the most important thing that he took away from it was the sense of camaraderie from the crew at Pixar at the time.
“I’d never really experienced that before, at that level, because it was such a loss that you didn’t need to have a meeting to explain how bad it was. People just knew. And both in the company but also in families and friends around us, and in the people in Point Richmond too. Maybe we got it back as good as we did, as close as we did, because of that. That’s a very emotional thing to say. It’s not technical.”
The thing that I take away about these experiences is that the spontaneity of the communal support speaks to the culture of Pixar the rest of the time. That kind of thing just doesn’t happen all of a sudden. You can’t have a disaster and instantly develop this kind of community and camaraderie.
It has to seep out. It has to be in the soil. You don’t just plant it and watch it grow in a day. It has to be cultivated over time, as it obviously was at Pixar.
Jacob agrees, citing the focus needed to restore the film. “To be there and not blink for 60 hours straight, while staying sharp, is effectively impossible. But suddenly you find food. You find a blanket, you find someone’s throwing you into a shower and taking you back out again. You’re like, “How’d that happen?!””
“It just worked out that way, without thinking about it. The lasting memory of the experience is the friendships that were formed through that. That journey together through that was one of the community binding together.”
“I’ll never forget ever being a part of Toy Story two. I was very lucky,” says Jacob. “I had that chance to work on a level of impact that helped keep Buzz and Woody, and “Toy Story” and the franchise, and Pixar all be a thing we talk about today.”

Java at NYU

So, I just finished up a semester of Java at NYU as part of their SCPS Java Certificate program. Taught by Sam Sultan, it was an amazing class. In fact, it was probably the best programming class that I have ever taken. I walked away after the ten weeks with a much deeper understanding of object oriented concepts. I recommend this class regardless of whether you want to study Java or not. It will give you a leg up with learning other languages.

Microsoft Surface

Microsoft is really pushing their new Surface tablet and touchscreen OS. I have been seeing constant ads on Hulu+ (I don’t have cable tv – Cut the Cord, Baby!!!), subway posters, and now they even have a kiosk set up in the Columbus Circle/Time Warner Building mall. There were eight employees running the small booth, I guess in preparation for a huge onslaught of curious passers-by.

In all fairness, though, I did take a spin through the Surface and it is actually really, really cool. I liked the execution of the keyboard but not the texture of it. It had a toothy, rubbery feel to it. Microsoft needs to come up with a better solution for the keyboard surface itself. Very cool idea though. Did Ballmer finally get something right???

Java Digital Magazine

With the help of Texterity, Oracle has begun publishing a glossy, bimonthly magazine focused on Java. From the mouths of Oracle, “Java Magazine is an essential source of knowledge about Java technology, the Java programming language, and Java-based applications for people who rely on them in their professional careers, or who aspire to.”

It is available as a free digital download for the iPhone, Android, and as a web based reader.