Offering potentially high-traffic hosting for existing independent wiki

[dantman]

I've been wanting to put my knowledge, experience, and experimentation to some practical use for awhile now (lots of info on how to sysadmin a high load wiki installation floating around in my head going to waste), so I approached my boss about it... he gave me the usual 'As long as it doesn't draw time away from Kommonwealth development or cause any significant expense' response, a little bland, but it's an ok.

So I'm thinking of offering some quality MediaWiki hosting to some NIWA wiki. I'm not thinking of the single box shared hosting kind of setup that I believe most NIWA wiki are currently running on, but rather the kind of multi-server load balanced installation that keeps a few high traffic wiki running smoothly. So I'm not really interested in the really small wiki, but rather the larger NIWA wiki that could use a setup that has more capacity to expand and handle large quantities of readers.

For pricing, I'm thinking of going based on server costs. Total up the costs for the month, perhaps adding something small for service ( you don't think your current shared hosting provider gives you anything less than a jacked up rate anyways), use analytics to guage the percentage of traffic going to each wiki, perhaps adding a bit of weight towards wiki which are using heavy extensions, and use that to divide the hosting costs among the wiki.

Going by the current cloud hosting we use and how a good install would be distributed among multiple servers, I'm guessing the starting setup would cost at least $60/mo (before we figure out how far we need to scale the number of nodes up to get good performance for the wiki) so I'm hoping at least 2-3 NIWA wiki are interested. Since it's based on cost, adding extra wiki doesn't increase the overall load much (the high base cost for a quality install is the number of servers you have to run, not the bandwidth, that's trivial), but the way it divides the cost between multiple communities means those communities have a drastically smaller amount they have to pay for hosting.

Now for the hosting itself... just to ease minds, I'll remind that I'm just offering the kind of hosting relationship any wiki's owner here already has with whatever company they're already paying to host their shared/dedicated/virtual/cloud hosting they use to put their MediaWiki install on, albeit perhaps a little more informal and personal in communication, and focused on managed MediaWiki rather than plain webspace. This isn't a WikiFarm idea where the host takes control.

Now the idea of me managing multiple MediaWiki installs over multiple-servers obviously does not lend itself to the idea of external people having direct access to the servers. The multi-server method complicates it, not to mention independent groups being able to mess up other wiki on the farm. Hence, I'd be the only one who could actually change something on the wiki's config. However if being able to see the source and suggest changes is important to everyone, I could try to work in a git repo or something that everyone could clone and send pull requests to get things added to the hosting besides what I do as requests.

If there's any wiki that wants to pursue Monaco that's also a possibility. I can try to finish up my monaco-port and see if I can make Monaco usable for any wiki that wants to use it. Later on any wiki that wants to pursue Monaco if they wish also has the option of having me try whatever it takes to recoup the hosting costs via advertisements if they don't feel like paying directly. I say Monaco simply because the only ideas I can come up with for sane ad placements that are a compromise between respectful placements in the wiki, and reasonable ad formats that will actually generate revenue are for Monaco. Of course, I do notice NIWA has some CC-BY-NC-SA wiki, I understand these aren't entirely compatible with that kind of offer so those are probably out, but the hosting offer still stands, the owners will just have to pay for hosting, there's no issue with NC there since the wiki's owners already have that kind of relationship with whatever company is providing them with webspace.

If you want some background on me and the company. The company is Redwerks Systems Inc., it's primarily a Web Design company, but we do a bit of programming (namely Kommonwealth, which also counts for some R&D), I'm the programmer there... if we had any other full-time employed programmers, I'd be the CTO (ie: You don't have to worry about whether I'm a junior in the company or not, the technical stuff centers around me). As for my background, I was a MediaWiki developer, I contributed various features and improvements to MediaWiki's core, and I've developed a few extensions, and contributed significantly to a few other big extensions. I use past tense because I've stopped doing big MediaWiki development and mulled into more JS stuff and interests in sysadmin stuff. So I know MediaWiki quite well, and information on how to manage a large MediaWiki network has been pounding it's way into my head. I have a bad habit of not being able to stick with one project, ie: I get bored and switch to another project to get a different taste... In this regard, working on our company's project (Kommonwealth) and MediaWiki hosting is a very good thing. With two projects in front of me, in two completely different languages, two completely different tastes, it will be easy for me to switch back and forth and be more motivated towards both since I'll always have something to switch to before switching back to keep myself from getting bored.

Now as for the practicality and evaluation of the idea. For anyone actually interested in having me do the hosting I think the best way to approach it would be to set it up, transition interested wiki over to it and tweak it so it runs smoothly, and run it for a month... at the end of the month we can tally up a guess on how much it would cost the wikis to continue to be hosted like that and everyone can decide if we want to continue that way. If not I can help everyone migrate back to their old hosting (don't drop out due to performance hiccups though, those can be ironed out by scaling the hosting usually). We can also delay major changes like MediaWiki updates to individual wiki till that's decided as well to make migrating back easier.

And finally, yes, I have a habit of writing long things occasionally (not too many, mostly intro stuff like this)... perhaps I should have given myself an entire night's sleep before writing this... sleepiness doesn't seem to affect my ability to think in any way, but does seem to affect my ability to write clearly and concisely (ie: when I write sleepily, my technical blurbs tripple in size)

[tacopill]

out of curiosity, if they exist and are already independent, what would you be offering that they already don't have?

[Jake]

Now the idea of me managing multiple MediaWiki installs over multiple-servers obviously does not lend itself to the idea of external people having direct access to the servers. The multi-server method complicates it, not to mention independent groups being able to mess up other wiki on the farm. Hence, I'd be the only one who could actually change something on the wiki's config. However if being able to see the source and suggest changes is important to everyone, I could try to work in a git repo or something that everyone could clone and send pull requests to get things added to the hosting besides what I do as requests.

No offense, but I would never run a site on a server that I can't modify myself. It just doesn't make sense. I don't want to have to go through a third party to install extensions or make configuration changes. Using version control as a workaround isn't exactly an elegant solution either. Isn't it possible for you to offer (S)FTP access to just the folder the wiki is located in? I can even do that with my shared hosting account.

[Axiomist]

To respond in short, I think your idea/plan could be a positive service around here. In the wake of all of the Wikia migrations, the wikis we haven't heard from such as Golden Sun, etc could be approached. I'm wondering if there's any more slack on your end about the price. None of the Wikia migrants are accustomed to paying for hosting. Would you still host a wiki if in lieu of payment, they allowed you to control the ad account?

[dantman]

out of curiosity, if they exist and are already independent, what would you be offering that they already don't have?

• Having me do the work installing extensions, configuration, etc... and monitoring the performance letting them focus on actually administering the wiki, instead of the software (not everyone is particularly technically experienced anyways).
• Direct help (or rather having me do it instead of having to figure out how) setting up technical stuff that the current people managing the software might not know how to do. (eg: There are plenty out-of-date installs, or ones that need help with pretty urls, etc... I can help configure the wiki -- and avoid SEO losses from path changes, something others might not consider -- and keep the installation up to date, including considerations
• A degree of my MediaWiki development experience... ie: Developing small extensions for things needed on the wiki, porting a skin the Wiki might have wanted to keep when they left a WikiFarm, etc...
• But mostly, a multi-server install that has the potential to handle levels of reader traffic that shared hosting wouldn't be able to cope with efficiently.

Now the idea of me managing multiple MediaWiki installs over multiple-servers obviously does not lend itself to the idea of external people having direct access to the servers. The multi-server method complicates it, not to mention independent groups being able to mess up other wiki on the farm. Hence, I'd be the only one who could actually change something on the wiki's config. However if being able to see the source and suggest changes is important to everyone, I could try to work in a git repo or something that everyone could clone and send pull requests to get things added to the hosting besides what I do as requests.

No offense, but I would never run a site on a server that I can't modify myself. It just doesn't make sense. I don't want to have to go through a third party to install extensions or make configuration changes. Using version control as a workaround isn't exactly an elegant solution either. Isn't it possible for you to offer (S)FTP access to just the folder the wiki is located in? I can even do that with my shared hosting account.

I'm trying to offer something that can cope with larger levels of readers (not really aiming at the wiki small enough that shared hosting has absolutely no issue on), that means multiple servers. At that level it's not a simple case of the files being on one server... On top of that the configurations that shared hosting usually use in order to make giving clients raw ftp access secure for other clients on the same server is usually the cause for some performance limitations as well as the things that make it hard to actually configure things like pretty urls. Plain filesystem access also makes it very hard to keep track of who broke what and how to fix it.

To respond in short, I think your idea/plan could be a positive service around here. In the wake of all of the Wikia migrations, the wikis we haven't heard from such as Golden Sun, etc could be approached. I'm wondering if there's any more slack on your end about the price. None of the Wikia migrants are accustomed to paying for hosting. Would you still host a wiki if in lieu of payment, they allowed you to control the ad account?

I'm actually looking to start with the ones that have already migrated away from a farm. Migrating from a WikiFarm requires extra work compared to migrating an already independent install. I think it's important to first ensure that we have setup a stable wiki hosting setup before doing that extra stuff, it's also best if the first people on the hosting actually have the ability to easily revert to the hosting they had before (it's harder to return to a WikiFarm). Also I don't like dealing with username migration issues (ie: people impersonating an imported user, and dealing with username confirmations and whatnot) when it comes to wikis migrating from a WikiFarm I'd strongly prefer to develop an extension that would allow the imported usernames to be reserved and have a special interface to allow those users from the old wiki to confirm their identity automatically without any question as to if they stole the nick or not. That would of course take development time (as would potentially whatever it takes to import images), which I CAN do while hosting other wiki, but wasting time on it before doing any hosting, when it's still questionable as to whether or not the extension will ever be used is a little harder to justify. So I'm hoping to start out with existing independent NIWA wiki, after that is stabilized we can find ways to help other wiki leaving WikiFarms to migrate and host them.

I did mention advertising a little. But again, that stuff takes extra development time and experience (really, do you think Wikia made any reasonable amount of return from those click based skyscraper ads it had back when Monobook was still default?), so yes, I'd hope to offer something like that as a possibility for those who want it, but I think it's best to take things a step at a time and do that after the ability to migrate, and more importantly, the stability of the hosting itself is ensured.

[Adam]

It's an interesting offer, and sounds like a much more scalable solution in future than much of our members' present hosting. For example, Porplemontage currently is hosting Mario Wiki, Smash Wiki, Pikipedia, Donkey Kong Wiki and Halopedia on a single box (or possibly a cluster). Others such as WiKirby and Lylat Wiki pay for standard shared hosting, while Zelda Wiki and Metroid Wiki remain on the servers of the sites from which they formed (Zelda Universe and ZeldaInformer respectively).

While I'd imagine most current NIWA members are happy with their current hosting situation, our ever growing list of affiliates may well include interested parties. Out of interest, would you be amenable to payment in advance on a month-by-month basis (rather than a commitment for any fixed period)?

My slight concern is that with the initial price tag you've pitched at, your offer is likely to appeal mainly to established independent sites, however the aspect of having no server-side control would be somewhat of a step down from their current level of access. This managed setup would be more appealing to those already operating within such environments, such as Wikia leavers, however they're not accustomed to paying for hosting at all (as already mentioned above).

Also I don't like dealing with username migration issues (ie: people impersonating an imported user, and dealing with username confirmations and whatnot) when it comes to wikis migrating from a WikiFarm I'd strongly prefer to develop an extension that would allow the imported usernames to be reserved and have a special interface to allow those users from the old wiki to confirm their identity automatically without any question as to if they stole the nick or not. That would of course take development time (as would potentially whatever it takes to import images), which I CAN do while hosting other wiki, but wasting time on it before doing any hosting, when it's still questionable as to whether or not the extension will ever be used is a little harder to justify.

Just for information, such a solution has already been devised and used successfully by the recently migrated Wowpedia. Having used the MediaWikiDumper PERL scripts to import the usernames, they allowed users to reclaim their accounts by creating a User Migration Extension.

[Jake]

I'm trying to offer something that can cope with larger levels of readers (not really aiming at the wiki small enough that shared hosting has absolutely no issue on), that means multiple servers. At that level it's not a simple case of the files being on one server... On top of that the configurations that shared hosting usually use in order to make giving clients raw ftp access secure for other clients on the same server is usually the cause for some performance limitations as well as the things that make it hard to actually configure things like pretty urls. Plain filesystem access also makes it very hard to keep track of who broke what and how to fix it.

I see, that makes sense. Repairing such a large wiki is bound to take a lot of work, so I can see why you'd prefer to do everything carefully. After all, it's always better to be safe than sorry.

[dantman]

While I'd imagine most current NIWA members are happy with their current hosting situation, our ever growing list of affiliates may well include interested parties. Out of interest, would you be amenable to payment in advance on a month-by-month basis (rather than a commitment for any fixed period)?

I'm not entirely clear on what you mean by "advance on a month-by-month basis" in relation to what I'm already thinking of so far. Fixed price tag? Commitment?

My slight concern is that with the initial price tag you've pitched at, your offer is likely to appeal mainly to established independent sites, however the aspect of having no server-side control would be somewhat of a step down from their current level of access. This managed setup would be more appealing to those already operating within such environments, such as Wikia leavers, however they're not accustomed to paying for hosting at all (as already mentioned above).

What are the most important reasons why having server-side control is important? The ability to make tweaks and check that they work on the site as you edit? The ability to see the code, and make changes (just that ability, not necessarily change it right away)? ... How often do you even want to make a change to the software running the wiki in a short period of time anyways?

There are various things I could try in order to give the ability to deploy code. But each thing I add reduces my ability to prevent people from shooting themselves in the foot (much easier when distributed among multiple servers with complications you may not understand), and worse, accidentally shooting someone else's wiki in the foot.

Also I don't like dealing with username migration issues (ie: people impersonating an imported user, and dealing with username confirmations and whatnot) when it comes to wikis migrating from a WikiFarm I'd strongly prefer to develop an extension that would allow the imported usernames to be reserved and have a special interface to allow those users from the old wiki to confirm their identity automatically without any question as to if they stole the nick or not. That would of course take development time (as would potentially whatever it takes to import images), which I CAN do while hosting other wiki, but wasting time on it before doing any hosting, when it's still questionable as to whether or not the extension will ever be used is a little harder to justify.

Just for information, such a solution has already been devised and used successfully by the recently migrated Wowpedia. Having used the MediaWikiDumper PERL scripts to import the usernames, they allowed users to reclaim their accounts by creating a User Migration Extension.

Interesting, I might use some of that as a reference. I was thinking of something a slight bit more flexible, and a little more user friendly when it comes to notifying the user, also wouldn't require using a script to generate disabled users. *cough* Also committed to the standard svn repo where everyone can find it when they look for something like it.


Should I poke some of those users about this discussion to see what they think?

[Zyeriis]

Is there no way you would consider aiding a wiki that needs to leave Wikia as soon as possible?
I represent the FFXIV Wiki and I recently joined the discussion about forming "SEIWA", the Square Enix equivalent to the NIWA. My concerns now focus on getting the hell off of Wikia, as Oasis completely wrecked my site. However, I don't want to move, just to move again, when the SEIWA idea comes to fruition.

I am perfectly fine with Ads, so long as they aren't audio/video/pop ups (you know, the truly annoying ones?) As others have said, those of us wishing to leave Wikia, are accustomed to not having to pay for hosting. I would like to keep things this way if at all possible through the aforementioned Ads.

Other than that, I personally would prefer a way to have SEIWA wikis, able to share hosting (or something else), to enable static usernames. This might be possible with other options but this way seems to be the simplest manner in which to do so (shared hosting).

So, I'll ask again, is there any way you can work my wiki: http://ffxiv.wikia.com in its departure from Wikia, and set it up in a manner that it would be able to help form "SEIWA", without having to use direct payment options (Ads are fine)?

[dantman]

Is there no way you would consider aiding a wiki that needs to leave Wikia as soon as possible?
I represent the FFXIV Wiki and I recently joined the discussion about forming "SEIWA", the Square Enix equivalent to the NIWA. My concerns now focus on getting the hell off of Wikia, as Oasis completely wrecked my site. However, I don't want to move, just to move again, when the SEIWA idea comes to fruition.

I am perfectly fine with Ads, so long as they aren't audio/video/pop ups (you know, the truly annoying ones?) As others have said, those of us wishing to leave Wikia, are accustomed to not having to pay for hosting. I would like to keep things this way if at all possible through the aforementioned Ads.

Other than that, I personally would prefer a way to have SEIWA wikis, able to share hosting (or something else), to enable static usernames. This might be possible with other options but this way seems to be the simplest manner in which to do so (shared hosting).

So, I'll ask again, is there any way you can work my wiki: http://ffxiv.wikia.com in its departure from Wikia, and set it up in a manner that it would be able to help form "SEIWA", without having to use direct payment options (Ads are fine)?

I can't be sure of the quality of the hosting and it's reliability matched with how much it costs to run until at least one wiki is using it... So it's fairly important for the first wiki I try hosting to be ones that have the capability to return to where they were before I tired hosting.

Ads and the only ideas I can come up with for reasonable ad placements, as well as getting techniques ready for migrating wiki take development time before they're ready. Ads also take time before they actually start earning enough to cover costs.

I don't believe I have the resources to indefinitely host ad-only wiki till the ads start making enough to cover the server cost without anyone paying directly for the hosting. Of course if I already have directly paid wiki on the hosting, it's only a matter of the time till I have the stuff developed to start taking in moving wiki that want to use the ad model.

By the way, I caution against sharing users across a set of wiki, unless the sysops on each wiki are trusted on the other wiki, sharing users opens up holes that allow sysops from one wiki to attack users in a way that affects the other wiki in certain ways. Sharing users across untrusted wiki is not actually an officially supported configuration. Also, shared users don't work so well if you aren't sharing a single base domain.

[Zyeriis]

I can't be sure of the quality of the hosting and it's reliability matched with how much it costs to run until at least one wiki is using it... So it's fairly important for the first wiki I try hosting to be ones that have the capability to return to where they were before I tired hosting.

Well, if it fails, it fails. If I remain on Wikia, well, then there wouldn't be anything to visit. Like I said, my entire wiki is completely destroyed by Oasis, there is absolutely no way to salvage things. So I just want to move and be done with it. If it doesn't work in the long run, so be it but as things are right now, the wiki has no future. Though, again, I would not be the only wiki in this move. To support the SEIWA idea, I wanted to have at least 2 wikis from Wikia move simultaneously, and function together to show that wikis who want to leave Wikia, can do so.

Ads and the only ideas I can come up with for reasonable ad placements, as well as getting techniques ready for migrating wiki take development time before they're ready. Ads also take time before they actually start earning enough to cover costs.

I am not entirely sure what you mean regarding the migration itself. Mediawiki exporting and importing should be more than sufficient. Admittingly, I don't know how troublesome it would be to get Mediawiki up and running but that seems to be the only roadbump in the actual migration. As for initial costs, those aren't that big of a deal, anything long term or expensive, would be a different story.

I don't believe I have the resources to indefinitely host ad-only wiki till the ads start making enough to cover the server cost without anyone paying directly for the hosting. Of course if I already have directly paid wiki on the hosting, it's only a matter of the time till I have the stuff developed to start taking in moving wiki that want to use the ad model.

Well, thats the one problem the SEIWA idea is facing. We want to make/have to make the SEIWA idea work for wikis leaving Wikia from the start, as the only independent Square Enix wikis I know of, are the ones that have the same topic as mine (we're "competitors" in their eyes) so placing one of them at the center would inevitably force my wiki from being able to join the SEIWA ("Conflict of Interest").

By the way, I caution against sharing users across a set of wiki, unless the sysops on each wiki are trusted on the other wiki, sharing users opens up holes that allow sysops from one wiki to attack users in a way that affects the other wiki in certain ways. Sharing users across untrusted wiki is not actually an officially supported configuration. Also, shared users don't work so well if you aren't sharing a single base domain.

I don't see how this is an issue with sharing users amongst an organization like the SEIWA, if the admins aren't trustworthy, then they probably wouldn't get into it in the first place.

[dantman]

I can't be sure of the quality of the hosting and it's reliability matched with how much it costs to run until at least one wiki is using it... So it's fairly important for the first wiki I try hosting to be ones that have the capability to return to where they were before I tired hosting.

Well, if it fails, it fails. If I remain on Wikia, well, then there wouldn't be anything to visit. Like I said, my entire wiki is completely destroyed by Oasis, there is absolutely no way to salvage things. So I just want to move and be done with it. If it doesn't work in the long run, so be it but as things are right now, the wiki has no future. Though, again, I would not be the only wiki in this move. To support the SEIWA idea, I wanted to have at least 2 wikis from Wikia move simultaneously, and function together to show that wikis who want to leave Wikia, can do so.

My base (conservative) estimate will run me $60+/mo, and that's assuming that a base 3 512MB/RAM cloud servers (Varnish, Apache+Memcached, MySQL) will be enough to stably run the wiki. Any necessary expansion to handle load efficiently will of course up the cost. That's also assuming I don't need to setup a dedicated instance for management tools and monitoring and can re-use our company infrastructure for that (if I have to open up the server for other people to have access to, or use a different host, then I'll have to add another $20+/mo server onto that as well...) the base cost for quality hosting is high, so I need to ensure from the start that I'm at least offsetting those costs.

If I don't then this could basically end up with; Me doing some development work to enable SEIWA wiki to migrate. And after I've finally done that dev work likely on a costly setup, setting up the independent wiki. Then after under a month finding I need to increase the number of servers to keep it stable, while finding that the advertisements aren't covering enough to offset the costs. And since the SEIWA wiki have no original hosting to return to, ending up having to completely drop the wiki leaving them dead sites.

Ads and the only ideas I can come up with for reasonable ad placements, as well as getting techniques ready for migrating wiki take development time before they're ready. Ads also take time before they actually start earning enough to cover costs.

I am not entirely sure what you mean regarding the migration itself. Mediawiki exporting and importing should be more than sufficient. Admittingly, I don't know how troublesome it would be to get Mediawiki up and running but that seems to be the only roadbump in the actual migration. As for initial costs, those aren't that big of a deal, anything long term or expensive, would be a different story.

Wikia only offers history dumps. I also need to track down or come up with a way to import images. And users is another thing to deal with.

By the way, I caution against sharing users across a set of wiki, unless the sysops on each wiki are trusted on the other wiki, sharing users opens up holes that allow sysops from one wiki to attack users in a way that affects the other wiki in certain ways. Sharing users across untrusted wiki is not actually an officially supported configuration. Also, shared users don't work so well if you aren't sharing a single base domain.

I don't see how this is an issue with sharing users amongst an organization like the SEIWA, if the admins aren't trustworthy, then they probably wouldn't get into it in the first place.

Just pointing it out.

[Axiomist]

So to sum it all up, one wiki must pay $60 to begin the experiment and be able to go back to where they were prior to the experiment?

[dantman]

So to sum it all up, one wiki must pay $60 to begin the experiment and be able to go back to where they were prior to the experiment?

Just be able to. And it doesn't have to be one wiki, it just has to be enough altogether to cover the costs. And ya, in case we need to scale up to fast and no-one can afford it the wikis should be able to return to normal hosting (ie: they just need to already be independent, I can't start with wiki migrating from a wiki farm that we can't import the updates to the wiki back to).