21st Century Technologies – 2013

Whiteboard Diagram
This was a good conceptual overview. Aren’t you enlightened?

Spawn was much more ambitious than I originally thought. In building a streaming game service, we were building:

  • A board with 4 CPUs on it. This was divided into two groups of two, consisting of:
    • An embedded Windows computer (Application Program Unit) – This computer actually ran the game itself
    • An embedded Linux Computer (MPU – don’t remember what ‘M’ stood for) – This computer controlled all input and output of the Windows machine, as well as its network stack. It handled messaging from the Internet, simulated a game controller, etc.
  • A cabinet with 20 of these boards
    • Handled the network routing of the 40 computers inside
    • A server farm in the data center to provide disk space to hold the games themselves and the users’ home directories for saving games.
  • A Windows application which would connect to our service. The user would plug a game controller into the computer, and the application would translated that into network commands that would talk to an MPU. It would also have an interface to selecting games.
  • A Mac application (never got started) to connect.
  • An Android Java application that could connect directly to our service.
  • A game stick (like a Roku, Firestick, or Chromecast). This device would plug directly into the TV’s HDMI port, and use USB for power (and for diagnostics or software updates). It had buildin WiFi and Bluetooth stacks for controllers to connect to.
  • A remote control for the initial setup of the game stick.
  • A custom game controller designed for connecting to our game stick via WiFi or Bluetooth.

We wanted to build out 5 data centers in North America, with more around the world as we grew.

When I got there, the first prototypes of everything were being developed. I was QA and Release Manager, but had no staff, so the first thing I worked on was building all of this software. I had to learn how we built:

  • A Windows application (easy)
  • Software for the MPU (embedded Linux)
  • Software for the game stick (Building an Android OS)
  • An Android application
  • Software for the remote control

Building for the Game Stick was the hardest to build, and I actually had to learn how to debug the Linux Android kernel on the device itself to figure out what our build problems were (1).

It was bold. It was ambitious. It was expensive.

I started in September, and by January things appeared to be coming together. And then… our chief embedded engineer left. And other people were starting to leave. My boss cancelled my weekly one-on-one, and never bothered to reschedule it.

A few weeks later, he quit, announcing that he was moving his family to Hollywood so he could help his daughter’s acting/modeling career.

It was obvious we were in disarray. About that time, the General Manager called me in to his office, and told me, “I can’t promise you that you will have a job going forward. If I were you, I would polish your resume, and start looking. I would be happy to be a reference, and I am fighting to keep us going. But I have to be realistic.”

Huh. Well, that sucked.

About a week later, he announced that Game Stop, our parent company, had given us a deadline. We had to show whatever we had in June.

And that team pulled together. Everybody worked hard on building a tiny data center with one board (two nodes) that could play from a catalog of two games. We hand-built two game sticks, and got two controllers to work with them. I was left with not much to do because we were in demo/hacking mode, but the last week leading up to the demo, I tested the hell out of the system. It worked great, and I got to play Borderlands 2 for a while. While we were putting this together, though, we were informed that our lease was running out, and the last day in the office that we had been at was going to be Demo Day.

On that day, our execs went up to Game Stop headquarters in Dallas, while the rest of us waited to tackle any fires that came up.

The demo went flawlessly. None of the Game Stop execs could believe it. They took the game sticks from office to office, plugged them in, and played games. They could not make the system fail. The saved game worked flawlessly. The video was good. The audio was good. There was no perceptible lag. It could not have gone better.

This was not expected. They honestly thought we were going to fail, and from what I understood, had been ready to lay us off the next week. But evidently, there was some sort of contractual thing with this demo, and they had to keep at least the management of Spawn around because we delivered on our promises.

When asked if we could start the next phase, which was to actually build a real data center in Dallas for a “regional beta”, they told us that we were going to be delayed; Game Stop had to deal with the rollout of both PlayStation 4 and Xbox One, and that they did not have the capital to build out a hundred-million-dollar data center.

Meanwhile, back at the ranch, as soon as the demo was over, everybody packed everything up. The IT people were up for almost three days; they got everything done in time for the trucks to move all of it to the new office on Sunday.

When we got to the new office on Monday, nothing was up at all. We all started helping the IT people start to build out the network and the internal infrastructure, but there was only so much the rest of us could do. Once there was WiFi, I was there to help with lifting and sorting cables, but the rest of the time, I had nothing to do at all.

And that did not change. Game Stop started trying to shop us around to other companies, but I don’t think the world was quite ready for us. Google Stadia was years away, and Game Stop was worried that Steam was going to start streaming games. Nobody bit.

I still continued to go to the office, but there was no work to do at all. So I spent my time learning more about programming for IOS. Was very productive! But I had to start looking for jobs.

I don’t know how the initial contact happened for 21CT, but I got a job listing for a Technical Lead in my email. It looked much more like a Project Manager position than anything else. I did a phone screen (nothing remarkable there). After the phone screen, they called to setup a day of interviewing. Looking good…

… and the day before the interview, they got in touch via email and told me that they had had a restructuring, and that the position was no longer open.

I’ve heard that before. Sigh.

(1)Basically, the 3rd party developer left instructions on how to patch the software, build it, and deploy it. But I wanted to put the finished source code after patching into source control. We did not have access to the source code of the video driver we were using, which is annoying. When I built the system via patching, it booted fine. But when I put in source control, it crashed on launch. Turns out, the video driver was built with a source code control set of source that was modified. When I build the Linux kernel with patches, it was also built with source that was modified. When I pushed into source control, it was not, and the kernel refused to load drivers that were built from modified source if it itself was built with unmodified source.

One thought on “21st Century Technologies – 2013

  1. Pingback: Recruited By Tech – 21st Century Technoglies – 2013 – CodingNova

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s