Topic

What's been going on?

We always get subtle hints about what's been happening behind-the-scenes of Glitch, but I'd be very interested to know, for example, what delayed the test so massively. Was it refactoring? Bugfixing? Feature-adding? I think it would be cool to be able to follow the development process at a little more technical level, as I for one (and I'm sure I'm not the only one) am extremely interested!

Posted 23 months ago by RobotGymnast Subscriber! | Permalink

Replies

  • Dude, I wish I could too! Short of getting hired by Tiny Speck I'm not sure how to make that happen. I assume something's been going on with the "new storage servers" referred to in twitter.com/#!/isglitchopen... but what?? The curmudgeon in me wonders if it could be that server-side JS doesn't scale so well after all? (Though EtherPad rocked before Google stupidly killed it.) Anyway right now they don't even seem to have a list of devs and their bios anywhere I can find, but I can't wait for TS to start publishing whitepapers on this stuff...
    Posted 23 months ago by EgIantine Subscriber! | Permalink
  • I wouldn't be surprised if JS wasn't scalable; I don't usually hear of servers relying on it (though I don't listen much).
    Posted 23 months ago by RobotGymnast Subscriber! | Permalink
  • Look in the upper right of screen; it's no longer mote time, now it's game time! I wonder if that means.....
    Posted 23 months ago by teet Subscriber! | Permalink
  • The overall architecture is not really a secret: we could even post a diagram (and if you bug us enough, we probably will) but here's the way it works:

    * there are the "game servers", which are written in java and do the minimum number of things they can. Among other things: hold connections open to clients, send messages out and receive requests from clients, compute movement for NPCs, balance load amongst themselves, persist the world state to the storage servers, and load things from storage into active memory as required.

    * the "JS" — the javascript bits in which the game logic is expressed: everything from quests and badges to the health/maturity rules for trees, the time it takes to learn a skill, how much energy you should get from a coin or the effect of activating spinach. The javascript runs in an engine called Rhino — en.wikipedia.org/wiki/Rhino... — and, via APIs exposed to it from the Java, scripts the behavior of the game server and controls the whole world.

    * the "storage servers" which take the constant stream of state information coming from the game servers and persists it in case the game server dies (or everyone leaves a location, or we need to restart, or if one game server gets too crowded and needs to split itself up into multiple servers etc.)

    Each game server is constantly writing to three different storage servers (for redundancy) so every object in the world (the tomato in the blue bag in slot 7 of your inventory; the musicblock in the cabinet of some empty house, the leftmost butterfly in Marakesh Meadow) and every bit of the state of those objects (whether you've already milked that butterfly today and what its current position and rate of motion is, how many seconds are left in your High Jumper buff from the spinach you just activated, how much metal ore is left in that rock, etc.) — all of that exists in three different servers from which any game server that needs it can look it up.

    Each of these three is complicated in its own way: we have put thousands of hours into tools which let non-programmers generate javascript via complex web forms to edit the requirements for a project or define the effect of a verb; the game servers can be individually shut down while the game is running and all the connected players reconnected to other servers with almost unnoticeable interruption, and … well, there's a lot: we've been working on it for a while :)

    The problem the week before last was with a new design for the storage servers: data integrity is critical and we had to do some extensive testing to make sure that nothing ever gets lost or messed up. The new design, while it met the efficiency requirements we were shooting for, have some bugs. Those are now resolved, but we didn't want to take any risks until we were sure.

    In a totally separate incident, we lost most of our internal tools for a day (IRC, bug tracker, source repository, wiki, etc.) because of a hardware failure in a system totally separate from the game. That alone lost us some time, but it also prompted us to move everything to its own system, remove single points of failure and make sure that never happened again, which also took a lot of unplanned time.

    Those two things caused a cascading schedule "fuck up" which put us behind by nearly a whole week. But, we're recovered now and some of the bigger projects are coming to fruition. So, we'll be back to testing soon.

    Incidentally, players often complain of "lag". It seems like that, but it is generally not slowness in client-server communication and it is almost never slowness on the server: our javascript is just being used to control the java and it runs on mature, battle-tested technology, even if out implementation is all crazy and new. No, the performance issues are all in the client: we're pushing Flash really hard, but more to the point, we've been adding features like crazy for months without stepping back to optimize.

    We're about to get started on the optimizations now (and probably won't be inviting many people until we have some significant gains — the choppiness is really irritating and will probably turn new players off). But, we are confident that there are gains to be had: we've just been so busy adding new stuff that we haven't taken the time to profile it, find the bottlenecks and alleviate the CPU-hunger.

    Hope that helps!
    Posted 22 months ago by stoot barfield Subscriber! | Permalink
  • That's all very interesting, but I really want to know is what colour socks are you wearing today?
    Posted 22 months ago by shhexy corin Subscriber! | Permalink
  • Yeah, that's great! Thanks for taking the time to explain that. It's pretty interesting from a development point of view. Plus I'm super-proud that I didn't have to Google any of that (though I guess that was probably the intent with which it was written).
    Posted 22 months ago by RobotGymnast Subscriber! | Permalink
  • Oh cool! So we're tuning the super cool Volkswagen Beetle, so that it'll run better on the awesome highway. :)
    Posted 22 months ago by Ixirim Subscriber! | Permalink
  • Wow, overload.

    At least you found the silver lining and fixed your systems so they are better. Can`t wait to see the new stuff.
    Posted 22 months ago by Ani Laurel Subscriber! | Permalink
  • Thanks Stoot for the deets. For some reason I find this interesting and have oddly gained the ability to understand much of what you say. :)
    Posted 22 months ago by Mac Rapalicious Subscriber! | Permalink
  • Thanks for letting us know what's been having on the dev side Stoot!
    Posted 22 months ago by nightning Subscriber! | Permalink
  • stoot, thanks for that awesome post! I have to concur with others in that I love reading about the technical aspects of what's going on. :)
    Posted 22 months ago by Rykos Subscriber! | Permalink
  • "we could even post a diagram (and if you bug us enough, we probably will)"

    Can I kick off the bugging? I love diagrams...
    Posted 22 months ago by dopiaza Subscriber! | Permalink
  • Officially bugging for some diagrams. :)
    Posted 22 months ago by en Subscriber! | Permalink
  • Diagrams are my cocaine.
    Posted 22 months ago by RobotGymnast Subscriber! | Permalink
  • +1 for a diagram
    Posted 22 months ago by Alex Subscriber! | Permalink
  • Diagrams. Yes. I also loved to hear the technical side of the development, since I live and breathe the technical.
    Posted 22 months ago by Orios Subscriber! | Permalink
  • Knock knock. Who's there? Diagram? Yes please!
    Posted 22 months ago by Hburger Subscriber! | Permalink
  • I'm still waiting to find out about the socks
    Posted 22 months ago by shhexy corin Subscriber! | Permalink
  • Eglantine: you can find a list of everyone who works on glitch, including all the devs, here: alpha.glitch.com/about/
    Posted 22 months ago by Bees! Subscriber! | Permalink
  • Flash 10.2 was released today.
    Posted 22 months ago by Tingly Claus Subscriber! | Permalink
  • Bees: annapeee isn't on there! :'(
    Posted 22 months ago by Hburger Subscriber! | Permalink
  • yes she is! second section!
    Posted 22 months ago by Bees! Subscriber! | Permalink
  • heh, this just popped up on FB: www.flickr.com/photos/stewa...
    Posted 22 months ago by Nanookie Subscriber! | Permalink
  • O stoot, u so crazy :)
    Posted 22 months ago by Hburger Subscriber! | Permalink
  • HOT DEV SOCK ACTION
    Posted 22 months ago by glum pudding Subscriber! | Permalink
  • Yeah, I was just coming back here to post that!

    shhexycorin, the answer to "what colour socks are you wearing today?" is the same answer everyday: ALL THE COLOURS OF THE RAINBOW!
    Posted 22 months ago by stoot barfield Subscriber! | Permalink