其它工具 Push - Reddit 代码部署工具

scmroad · 发布于 2012年8月24日 · 269 次阅读
96

Reddit开源了自己的代码部署工具,就叫push 他们没多少人,服务器170台左右,这工具看上去也有点简陋。但是发布稿里这句话酷毙了:“实在想不出什么理由不开源。

We deploy our code to the ~170 application servers currently in our infrastructure via SSH and Git. This may or may not be useful to anyone else but we like to think that there has to be a compelling reason not to open source code, so here it is in all its glory. https://github.com/reddit/push

共收到 3 条回复
4
laofo · #1 · 2012年8月24日

好消息不断啊,最近开源了很多很不错的工具,这下子有时间可以多学习学习了。

96
scmroad · #2 · 2012年8月24日

Is a deployment, which takes 45 minutes acceptable for you guys? I mean, why no Bittorrent on that size of environment? https://vimeo.com/11280885

[–]kemitche The average deploy is around 10 minutes. The roadblock isn't going to be getting the bits into place - it's the fact that we can't restart everything at the same time without overloading the apps and databases, so even if bittorrent sped that up, it's not the current deployment "bottleneck"

[–]spladug Right, as /u/kemitche said, it's not even remotely an issue of moving bits around fast enough. I mentioned above in quite a bit of detail what causes it to go slow, some components building on each server and deliberately slowing down when over-capacity, and BitTorrent would help neither of those situations. The latter is a situation we avoid being in by provisioning extra servers. The prior is solvable with a centralized build of a complete binary package, as is on the to do list in the README. It doesn't happen nearly frequently enough for it to take priority right now though.

[–]phira process occurs after automated tests are run? How often do you deploy? How long does a deploy normally take? Are there problems caused by the delay between servers? I presume this means some care has to be taken with the backend to avoid old versions being unable to operate on a new schema or api call or similar? What do you do if a deploy fails partially (network split makes some app servers unavailable or similar)?

[–]spladug[S]
Interesting. I presume this process occurs after automated tests are run? I'll go with yes, but only because we don't have any automated tests. :( How often do you deploy? As frequently as we have code to deploy. Some weeks that means only a couple of deploys, others it means 5 - 8 times in a day. How long does a deploy normally take? That depends drastically upon a few things. First of all, if the site is under a lot of load we'll try to avoid taking too many apps down at a time, so we'll increase the wait interval between each app (the sleep time). /u/alienth suggested the --shuffle option which allows us to move randomly through the app list instead of linearly which helps a lot as our app servers are arranged into pools; all of the comment pages are routed to e.g. apps 30 - 90. This is useful for us in a couple of ways, but the downside is that pushing linearly through the list would hurt a single pool inordinately at any given time. So when --shuffle's on, we can pretty much push without sleeptime even at peak these days. The other major factor is if any Cython files were changed or if we're pushing out new translations. Both of these currently trigger a makefile on each of the appservers which takes a few moments to build, which adds up en masse. That's why I have the comment in the README saying we should build once and package up for deploy. Current thoughts there are a Python binary egg or a debian package. Luckily those kinds of pushes are relatively rare so it doesn't bother us enough right now to warrant attention. In short, somewhere from 6 minutes to 45, with the average being around 10. Are there problems caused by the delay between servers? I presume this means some care has to be taken with the backend to avoid old versions being unable to operate on a new schema or api call or similar? Yes, this is definitely something to consider. We're not doing all-or-nothing deploys so we have to either push things out in stages or write the code in such a way that it gracefully handles both versions running at the same time. What do you do if a deploy fails partially (network split makes some app servers unavailable or similar)? If there are app servers that aren't able to be updated, we'd disable their nginx which will cause them to fall out of the haproxy after their next health check fails (a few seconds). This very rarely happens and instead the app is usually just hard down in which case we'll just kill it and replace it with a freshly built one that has the new code by default. Given that most pushes are only 10~ minutes, it's pretty rare for something like this to happen.

[–]phira

No tests? Gosh. How do you test then? Thanks for the answers!

[–]dvito

The same way my company does much of the time

96
scmroad · #3 · 2012年8月24日

Overview Welcome to push tool, I love you!

Configuration The tool will look for ini files in /etc/push.ini and ~/.push.ini. Options in the latter override those in the former. Most of them should probably be left the same by all users, but some (particularly the SSH configuration) could be useful to configure individually. See etc/push.ini.example for a complete example configuration.

Configuration is divided into a few sections:

defaults Some basic defaults for CLI options that can be overridden in user-configs.

sleeptime -- duration to wait between hosts. shuffle -- whether or not to shuffle the hostlist before deploy. dns A DNS zone transfer is used to grab a list of all hosts that can be pushed to.

domain -- the domain to do a zone transfer from. aliases Aliases are mnemonics for a commonly used set of hosts to push to. You can directly specify hostnames as part of a group, glob them, or include other aliases within an alias.

[aliases] apps = app-* job-* dbs = pg-* all = @apps @dbs ssh user -- the username to use when connecting to servers. key_filename -- which key file to use for ssh connections. strict_host_key_checking -- should ssh be strict about host keys? defaults to on. timeout -- how long to wait before timing out, defaults to 30 seconds. paths log_root -- the directory to store push logs in. wordlist -- where to find a list of words to choose push names from. deploy build_host -- which host to run build commands on. deploy_binary -- the root command to execute for deploy-commands. build_binary -- the root command to execute for build-commands. syslog Some basic notifications of deploys starting/ending are sent to syslog for tracking in external services.

ident -- the syslog ident to use for log messages. defaults to "deploy". facility -- the syslog facility to use. priority -- the syslog priority to use. harold Notification and status updates for deploys are sent to an IRC channel via our IRC bot Harold.

Configuration is done through Wessex's configuration file.

Future Plans for the future include:

move from SSH to MCollective. and better parallelization as a result build a complete package on the build server to avoid Cython/i18n recompilation.

需要 登录 后方可回复, 如果你还没有账号请点击这里 注册