How we build our web-based learning interface

When Codecademy was born in the summer of 2011, the natural choice for a frontend stack was Backbone + jQuery. Backbone was less than a year old and widely considered the state-of-the-art framework for creating frontend applications. Combined with jQuery, one could do a lot of novel things with the web browser. In our case, we built something that helped teach millions of people how to code, interactively, for free.

Versions 1 and 2 of our learning interface: a code editor and a "Run" button

Over time, the company's mission outgrew that initial product. Teaching people programming skills that make them employable involves a lot more than simply writing code and hitting "Run"; we realized we need an interface which can adapt to different kinds of content and evolve in complexity over time to keep up with our learners' advancing skills.

We wanted the ability to present learners with arbitrary combinations of tools ("components") as they go through the steps of building an application, exploring a technology, learning a command line tool, etc. Some of the components we currently have include a code editor, a terminal, and a web browser.

Try it to learn some basics of Rails or AngularJS!

Early concepts of an interface which can switch between different stages of developing an application
Actual implementation of the interface teaching a user how to generate a Ruby on Rails model and run a database migration (WIP)

Building this warranted a rewrite, and 3 years later we were faced with more options when it came to choosing a stack. It wasn't an easy decision. Ultimately, we decided to go with React.js with a Flux-like pattern on top. We also took the opportunity to redo our build process and preemptively adopt the next version of JavaScript.

In summary, our frontend stack recently made this transition:

2011-2014		2014-2015
Backbone	→	React
mustache	→	JSX
jQuery	→	much less jQuery
ECMAScript 5	→	ECMAScript 6
require.js	→	webpack

React

http://facebook.github.io/react/

We chose to use React because our application is extremely view-heavy; there is lots of rendering and state change going on as a user advances through our interactive lessons. This is what React excels at. There were also things factored in such as the ability to render everything server-side (something Angular.js cannot do, for example).

The team is very happy with it so far. I think we saved weeks in the original 3-month timeline for this rewrite simply by using it. We didn't have to write repetitive boilerplate code for DOM manipulation or view/state synchronization. The amount of work React does for you out of the box is a breath of fresh air.

And it seems there's constantly new, innovative things happening in the React ecosystem like React Native and React Canvas.

ECMAScript 6

http://kangax.github.io/compat-table/es6/

We also went from simply having ES6 "on our radar" to actually embracing it. We automated a simple rewrite of our codebase from using the require.js AMD API to using the new language standard, import and export. It feels good to know we're relying on the language itself rather than continuing to build the company's product in a dying framework and syntax.

The main benefit of adopting ES6 has been developer happiness; boiled down, it's simply a new layer of sugar on top of the JavaScript everybody is already used to. It's also nice to know our codebase is a little more future-proof.

Since ES6 is a purely additive update to the language spec, it wasn't too hard to automatically migrate everything! It took me 5 or 6 days of work before we were able to deploy the new build to production.

webpack

http://webpack.github.io/docs/

None of this would have been easy or fun without webpack. Our old build process was literally a custom node.js script, an ugly pile of hacky regexes and fs calls.

Using webpack together with babeljs we've been able to stop worrying about our build process altogether; the only thing we have to maintain is a simple config file.

The other important point is that webpack is fast. The local page load time for our app went from 30-40 seconds to 2-3 seconds. AMD means you're loading every .js file separately. With webpack, updates are rebuilt in roughly 100ms and we're then only loading one file on the frontend. It was easily an order of magnitude speedup in the feedback loop, which has made for very happy engineers.

Webpack is also designed in a very modular way, so playing with new technology like Flowtype is as trivial as installing a loader for it.

Summary

Many engineering teams are going through this right now, so I want to give a simple takeaway that might help a few people make their own decisions.

Technology	Why we chose to adopt it	Benefits we've seen
React	Building complicated single page app with big view hierarchy Need server-side rendering Reassured by strong project development momentum	Smaller code base Fewer bugs More performant app Faster development
ES6	Have to eventually :)	Writing JS is more fun
Webpack	AMD is slow Want to replace fragile, incomprehensible build process Want flexibility to try out new technology	10x local page load speedup compared to require.js Easy to link external libraries via npm Easy to integrate ES6 "transpilation"

Feel free to reach out if you have questions or want advice related to any of this. And it's worth mentioning that we're hiring, so reach out if you are interested.

Artur Sapek
Engineer, Codecademy
artur@codecademy.com

How I learned that the default settings are not production settings

I got the rare Linode Alert in my email today, claiming 111.8% CPU usage on my modest little 2048 box. When I followed the link to Linode's dashboard, I saw on my charts that it had actually risen to around 600% and had been at that level for about 3 hours. What the fuck? What am I doing that could possibly have gone so wrong all of a sudden? I serve a couple webapps on this box but they have always been well-oiled machines.

As soon as I ssh in and run top, I'm even more confused. User www, which as far as I know doesn't do anything at all, is throwing a party with a perl script called "m64."

I killed that shit as soon as I had collected evidence. I'm surprised and glad Linode didn't shut me down for clobbering their machine like that. Then I started Googling for this script name, to see what it could have been doing. The only thing I could find was this Feb 2014 post which indicates that it's a Bitcoin mining script. That makes sense, since one of the two things that box serves is my Bitcoin market data tool for traders. It probably drew some attention.

Naturally I was curious how someone got a file on my server. It was running as an unprivileged user, which was a slight relief, but nothing compared to the sinking feeling that my beloved Linode had a malevolent stranger creeping around in it. I went through the ssh logs at /var/log/secure and was horrified. In the last 12 hours, there were thousands of failed ssh login attempts mostly as root, admin, and apache, but also many others.

Literally tens of thousands of lines of this. A login request every second.

Jun  8 16:01:20 meowset sshd[15833]: Failed password for root from 117.21.191.210 port 2643 ssh2
Jun  8 16:01:22 meowset sshd[15841]: Failed password for root from 117.21.191.210 port 1382 ssh2
Jun  8 16:01:24 meowset sshd[15833]: Failed password for root from 117.21.191.210 port 2643 ssh2
Jun  8 16:01:25 meowset sshd[15841]: Failed password for root from 117.21.191.210 port 1382 ssh2
Jun  8 16:01:26 meowset sshd[15833]: Failed password for root from 117.21.191.210 port 2643 ssh2
Jun  8 16:01:27 meowset sshd[15841]: Failed password for root from 117.21.191.210 port 1382 ssh2
Jun  8 16:01:29 meowset sshd[15833]: Failed password for root from 117.21.191.210 port 2643 ssh2
Jun  8 16:01:29 meowset sshd[15841]: Failed password for root from 117.21.191.210 port 1382 ssh2
Jun  8 16:01:31 meowset sshd[15833]: Failed password for root from 117.21.191.210 port 2643 ssh2
Jun  8 16:01:31 meowset sshd[15841]: Failed password for root from 117.21.191.210 port 1382 ssh2
Jun  8 16:01:31 meowset sshd[15842]: Disconnecting: Too many authentication failures for root
Jun  8 16:01:31 meowset sshd[15841]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=117.21.191.210  user=root
Jun  8 16:01:31 meowset sshd[15841]: PAM service(sshd) ignoring max retries; 6 > 3
Jun  8 16:01:33 meowset sshd[15833]: Failed password for root from 117.21.191.210 port 2643 ssh2
Jun  8 16:01:33 meowset sshd[15834]: Disconnecting: Too many authentication failures for root
Jun  8 16:01:33 meowset sshd[15833]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=117.21.191.210  user=root
Jun  8 16:01:33 meowset sshd[15833]: PAM service(sshd) ignoring max retries; 6 > 3
Jun  8 16:01:36 meowset sshd[15961]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=117.21.191.210  user=root
Jun  8 16:01:37 meowset sshd[15940]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=117.21.191.210  user=root
Jun  8 16:01:39 meowset sshd[15961]: Failed password for root from 117.21.191.210 port 1931 ssh2
Jun  8 16:01:39 meowset sshd[15940]: Failed password for root from 117.21.191.210 port 2359 ssh2
Jun  8 16:01:41 meowset sshd[15961]: Failed password for root from 117.21.191.210 port 1931 ssh2

An obvious brute-force attempt, but not completely automated. These are interlaced with seemingly human attempts at logging in as users like shit, and shit2. Pretty futile and confusing.

Jun  8 16:34:55 meowset sshd[27949]: Invalid user shit from 195.158.109.185
Jun  8 16:34:55 meowset sshd[27957]: input_userauth_request: invalid user shit
Jun  8 16:34:55 meowset sshd[27949]: pam_unix(sshd:auth): check pass; user unknown
Jun  8 16:34:55 meowset sshd[27949]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=mail.sterling.com.mt
Jun  8 16:34:55 meowset sshd[27949]: pam_succeed_if(sshd:auth): error retrieving information about user shit
Jun  8 16:35:02 meowset sshd[27949]: Failed password for invalid user shit from 195.158.109.185 port 56864 ssh2
Jun  8 16:35:02 meowset sshd[27957]: Received disconnect from 195.158.109.185: 11: Bye Bye
Jun  8 16:35:04 meowset sshd[28011]: Invalid user shit2 from 195.158.109.185
Jun  8 16:35:04 meowset sshd[28012]: input_userauth_request: invalid user shit2

Here's a full list of invalid users for which login attempts were made, which I find really interesting. It looks like a lot of these are coming off Centos's standard users, which apparently my installation didn't include.

admin
ajay
ale
anti
asterisk
avahi
bszsimon
bwadmin
cacti
chem
cosmos
csaba
csanyip
cskiraly
cvsadmin
db2inst1
deploy
desiree
dev
devteam
garrysmod
gdm
gergely
git
gnats
guest
guest1
guest2
ielm
jboss
jessie
jim
jira
johnky
karakai
konsti
krejczinger
kris
lhc
lswang
magento
manager
manon
master
meres
mihaly
minecraft
mysql
nagios
nam
ndbao
news
nikos
no-reply
noreply
nouser
oracle
pappd
periodic
photo
photos
postgres
postgres2
postgress
postgrestest
postgresuser
postmaster
redmine
research
rpcuser
rsync
rtorrent
rvm
sabayon
share
shit
shit2
support
supports
supporttest
supportuser
temp
test
test1
test2
tom
tomcat
tsumura
upload
user
user01
user1
user123
user2
userfetch
userftp
userguest
userhome
users
usr
usr1
usr2
vwalker
william
www-data
xochitl
yuji
zabbix
zimbra
zsofi
zspandi

There were only a handful attempts for each of these. Apparently if an attacker detects that a box accepts ssh using a password, they try to log in as every possible user that could exist in the hopes of an easy password or an empty password.

Also, for fun, here are all the IPs that tried brute-forcing in as root. The majority of them seem to be Chinese.

1.93.33.77
100.2.156.3
113.171.10.1
115.239.248.61
116.10.191.163
116.10.191.170
116.10.191.196
116.10.191.232
116.10.191.234
116.10.191.253
117.21.191.210
122.225.103.118
195.158.109.185
211.25.206.34
220.177.198.26
220.177.198.43
222.186.40.170
222.186.56.33
61.142.106.5
61.153.105.69
62.212.141.128

Seeing all this made me a little anxious that perhaps they would manage to knock down the gates any minute. The ssh logs seem to be truncated, so I don't even know how long this has been going on for - relentlessly for all of today at the least.

But what about that Bitcoin miner? As the logs show, apparently many different IPs had managed to log in as www. I'm not even sure where that user came from, or what its password was.

Jun  8 10:54:03 meowset sshd[2083]: Accepted password for www from 23.102.134.179 port 1168 ssh2
...
Jun  8 14:39:51 meowset sshd[18941]: Accepted password for www from 191.238.227.219 port 1048 ssh2
...
Jun  8 15:03:28 meowset sshd[27443]: Accepted password for www from 191.236.20.93 port 1056 ssh2
...
Jun  8 16:36:25 meowset sshd[28548]: Accepted password for www from 195.158.109.185 port 33915 ssh2
...
Jun  8 17:55:00 meowset sshd[24431]: Accepted password for www from 23.96.51.35 port 1072 ssh2
...
Jun  8 18:37:27 meowset sshd[7806]: Accepted password for www from 23.96.53.244 port 1160 ssh2
...
Jun  8 19:42:04 meowset sshd[31758]: Accepted password for www from 23.102.134.202 port 1064 ssh2
...
Jun  8 19:46:13 meowset sshd[765]: Accepted password for www from 188.25.126.228 port 64247 ssh2

And the last successful login attempt was followed by the password being changed. Then shortly after, the miner started.

Jun  8 19:47:27 meowset passwd: pam_unix(passwd:chauthtok): password changed for www

The moral of this story

What did I learn today? Set up ssh keys! Then in /etc/ssh/sshd_config, change PasswordAuthentication yes to PasswordAuthentication no :)

I hope whoever that was enjoys the three cents they made today.

For anyone interested, you can download the file that was running. It's a binary. I'm stuck trying to learn more about it, but I'd love to hear it if you can. me@artur.co.