Neotel Hosting Services: Unprofessional and mediocre

After a dreadful 11 months hosting our business with Neotel and having exhausted all avenues with Neotel senior staff and executives, I have reached a point where our business relationship has become unworkable and while I typically do not like to post about business relationships having gone sour, I do think that based on Neotel’s senior staff attitude it is only fair to provide other companies an insight of what went wrong and to possibly look at other solution providers, as I do not believe that Neotel is capable of turning around their operation (or staff’s attitude).

How it all started

Neotel approached us in 2013 with a compelling business proposition, that hosting in their state-of-the-art data-centre will not only save us in hosting fees, but will also improve our uptime, protection from DDOS, provide us with extensive monitoring and after all, Neotel is backed by Tata. It was a relatively easy exercise to list our requirements as it was really a straightforward migration from our current hosting provider to Neotel and moving two racks of servers with a /24 IP-range is not really such a big process.

The pre-sales discussions with the sales staff and technical teams went well and a proposal was drafted and accepted within a few weeks and what comes next should have served as a fair warning, but I ignored all the alarm bells going off.

The shoddy project management

The account management and project management was a half-baked initiative. To be honest, during our 4 months preparation to move infrastructure and systems, our “dedicated” Neotel project manager met with us twice, sent 2 emails and went missing during the 4 weeks leading up to the implementation. Although this was concerning and was escalated numerous times (and brushed off / ignored by Neotel senior management), we went ahead with the move, as our own technical team is highly experienced in moving and migrating our infrastructure and our only dependency during the move on Neotel was really “just” to have access to our racks and networking being properly configured.

Our move was scheduled early January 2014 (from the 2nd – 5th) and it began with a rocky start, as Neotel managed to “misplace” our data-centre access and security refused us entry for several hours. It also did not help that none of the key staff was available (both the project manager and the dedicated account manager decided to be on leave during this week and kept their phones off).

I can not recall a single issue or request which was resolved within SLA’s or within an acceptable time-frame. In all cases, once we felt that we had waited reasonable long enough, we escalated to senior- and executive staff and even those escalations hardly ever resulted in an acceptable response.

Over promise and under deliver

Once we managed to settle in, the next 11 months turned into pure hell and not a week went by without running into support- or management-issues. Don’t get me wrong, our infrastructure and services functioned fine most of the time, but when we relied on our project manager, account manager or Neotel’s technical support team, it was always awkward as Neotel made us feel like they are doing us a favor and in most instances it felt that our reaching out was an inconvenience for them. The off-shore (Tata) call-centre also did not help, as the support staff was more interested in meeting SLA’s and closing tickets early than actually ensuring that our requests were resolved properly.

But when things went wrong, it went horribly wrong and Neotel technical support was not capable of providing a speedy resolution. We typically have a 99,9% uptime, but when the shit hits the proverbial fan, it is not unusual to run for several days with issues – like the one in September, where Neotel had issues with upstream providers (but was not aware of it until we notified them):

One of our primary reasons for the move was that we were guaranteed that Neotel has a state-of-the-art DDOS infrastructure and we learned the hard way, that Neotel actually had nothing in place. It took 5 (!) months after go-live to arrive at the point where Neotel conceded that we could use Arbor Networks via Tata Communication for DDOS mitigation which would require Arbor in a mitigation scenario to first phone us, then we had to phone Neotel, then had to phone Arbor back and mitigation would only become effective within 1 hour (according to their SLA’s). This was clearly not what we initially discussed and was obviously not a viable solution for us. It was surprising that Neotel lacked the experience and capabilities in this area. It was also frustrating for us, as this was one of the key value-propositions for the move of hosting providers.

The highlights of non-delivery and a lack of processes / ownership:

• The go-live of our migration took actually 3 months, as Neotel could not configure our firewalls and it took 3 months of escalations to senior management to eventually make a trivial change.
• As part of the service offering, we were supposed to have full access to monitoring of the firewalls, which never materialised.
• A change request to have our VPN access configured took 2 (!) months to resolve.
• Despite dual-power feeds, one rack dropped all it’s power and it took Neotel over 1 hour to figure out that one of their circuit-breakers tripped. Weeks later we received a post-mortem report, that the circuit breaker in the rack tripped resulting in complete rack-failure, which was concerning, as all racks were supposed to have independent dual-power feeds and the secondary circuit should have continued to provide power.
• During a server outage and despite pre-arranged access to the data-centre, data-centre security staff refused access of our engineering staff for 3 hours. We never received an explanation why this could happen other than “corrective measures will be put in place to avoid a re-occurrence”
• Over a period of 2 months our monitoring servers reported “flapping” connections and our websites could not be contacted from regions outside of South Africa. Eventually the problem “disappeared” and although a post-mortem was requested, the only response we received was “we believe our upstream providers had an issue and this seems to be fixed now”.
  

  Frequent international “jitter” with long lasting outages on Neotel network

• The reporting infrastructure is completely inadequate:
  • Power consumption is reported on a per-feed level and it is not possible to report on per-rack or per-site level. In our case (2 racks, 4 power feeds per rack) we had to manually consolidate 24 CSV files (12 CSV files per rack) to arrive at site utilisation. For a period of time, Neotel data-centre staff had to manually (!) read electricity meters to provide consumption figures.
  • Bandwidth utilisation is reported on a per-site basis. There is no break-down of inbound and outbound traffic. It is not possible to report on a per-IP basis or virtually group a number of IPs – this makes it impossible to do any capacity planning and we had to rely on our own reporting.
  • The provided Infovista portal did never work, which was actually a good thing, as the way it was configured would have been very cumbersome to use.
  • There seems to be no alerting-/notification infrastructure in place to alert customers on outages. In all instances our own alerting picked up core-network outages before even Neotel engineers were aware of it.

Missing the essentials and the billing crisis

Neotel’s customer portal was supposedly “state-of-the-art” and once we eventually got access to it, we were shocked that a hosting provider of such size missed the most obvious. I was even more surprised, that the issues below had been in place for years and no other hosting customer seem to have bothered to raise or question them:

• Registering on the Neotel portal was flawed with errors. Incorrect telephone number validation, then broken registration links. Logins to the portal would just expire, resulting in cumbersome password reset, where it took a really long time to receive the password reset email.
• Invoices for the first few months went “missing” and had to be reloaded
• Out of 10 invoices received, 8 were incorrect.
• In some instances it looked like services where manually charged and were actually not applicable to our account (i.e. one month we were charged for “Voice Usage”)
• Invoices had spelling mistakes (such as “Neotel Bustable Data”) or displayed wrong VAT amounts
• For 9 months Neotel could not report bandwidth or power usage. AT ALL. Yes, I need to repeat this: Neotel’s mechanism to measure utilisation was wrong and all bandwidth reports were in-accurate and unusable
  

  Neotel could not report on bandwidth utilisation for 9 months

  The above picture should be telling: Our business utilises up to 10TB of data per month and Neotel reports showed usage of up to 90GB. We were then told that the label of “GBs” is wrong and it should actually be “TBs” which was obviously also incorrect.

• Power utilisation reporting only started working in August and Neotel was not able to provide us with historic data.
• To date, Neotel is incapable of reporting bandwidth utilisation on a per IP-level and has indicated that this will not be possible.
• Billing in general is such a mess, that Neotel has on record, that out of the two racks we are hosting with them, the one is commissioned in January and the other in May. Obviously another mystery to us, as both racks where populated and taken online in January and have been billed with a start date of January 2014. We tend to overlook those issues, as explanations to those issues are hard to come buy – for example our firewall was provisioned in January but according to billing records was only taken online in May.

As you can expect, if a hosting company like Neotel is not capable of getting the basics (collect and report on utilisation of power and bandwidth) right, that this will ultimately also affect your billing. For the first 9 months Neotel was not able to bill us for bandwidth and only after several escalations to senior management, did the company decide to investigate and found that all customers had been billed incorrectly and the reporting issues was not just isolated to our account but a company wide issue.

Neotel eventually furnished a letter in October 2014 that their usage data-collectors did not measure usage correctly and based on our experience, has been going on from at least January 2014, but I personally think that the issue had been present for substantially longer.

 

The attitude and lack of professionalism

When a company like ours decides to host our infrastructure with a hosting company, it is important for us that we work together as partners. Unlike many other businesses, our marketplace is a platform for thousands of sellers making a living and relying on our dependable services. Although customer service is generally poor in a consumer environment, I have never experienced such patronising and arrogant attitude as from senior and executive staff from Neotel.

Just the arrogance shown is astounding. My question why Neotel is not able to provide bandwidth utilisation on a per IP-basis, was laughed off by their senior network architect in an escalation meeting in September with the comment “You are the first customer requesting this information. No-one actually needs this”. We were also told that we should be “grateful”, as Neotel will be “lenient” and not charge us for the previously understated invoices (even if they wanted, they couldn’t as Neotel collected the data wrong in the first place).

While it is perfectly understandable and acceptable that mistakes happen in IT, it is certainly not acceptable if a hosting company such as Neotel does not accept ownership and accountability for their poor performance and it has gone too far when customers get patronised and belittled in meetings. I honestly have never experienced this level of communication in a corporate environment and that in itself should be warning enough that the company staff does not value their customers enough and lack any willingness to work on maintaining long-lasting business relationships.

Where from here

Despite more than 10 months of escalations and attempts to resolve the various service issues, we feel that Neotel needs to fundamentally change staff attitude and take ownership of their problems. Unfortunately it is not as easy as just replacing some staff, as the issues we faced had been escalated to senior- and executive management and the level of complacency, ignorance and arrogance is present on all management levels within the company. I do think that should the Vodacom buyout happen, this will be positive for Neotel, as Vodacom will hopefully realise very soon that Neotel lacks structure and organisation and will implement the necessary measures to make the company work.

Although Neotel’s realisation that they had under billed their customers for a long time might have some upsides (such as improved revenue), it will certainly have negative effects with customer churning, as their future invoices will be substantially higher than in the past. I am sure that the realisation of understated invoices has now resulted in an adjustment of service costs which will probably mean that Neotel will become less competitive with their offerings.

It is also puzzling to me as a hosting customer, that no other client of Neotel has ever questioned the lack of reporting and the accuracy of data. It was very obvious that bandwidth utilisation of a few hundred GB per month across two racks was incorrect, but yet we were told that no other customer ever complained. I struggle to understand this, since capacity planning is an important aspect for any customer and as such it is crucial to actually understand usage.

I even less so understand that Neotel did not manage to pick those issues up. Surely if a company like ours consumes in excess of 8TB of data (which Neotel would have to pay to one of their upstream providers), but Neotel themselves only bills us 80-90GB, someone in their finance division is not doing a good enough job and has never bothered to question the discrepancy between bandwidth paid to upstream providers vs. bandwidth billed to customers.

We have served notice of termination of our account and will move to another hosting provider in the next 2 months. The upcoming move is exiting for us, as we will finally be able to actually get insight into our capacity utilisation (something we had to implement ourselves for now due to the lack of Neotel’s capabilities) and we will not have to waste hours per month dealing with administrative and operational issues due to Neotel’s lack of structure and organisation.

I personally strongly advise any company thinking about moving to Neotel hosting, to rethink their move or at least ensure that all their requirements can be appropriately managed by Neotel. None of our requirements where out of the ordinary, and I do not think that Neotel’s slogan “The Neotel Way is a path tailored to suit your needs. It is innovative, world-class, and solutions-driven.” was ever appropriate for us.

I urge all existing Neotel customers to review their invoices, usage and billing and since there does not seem to be any tangible improvement in sight, suggest to move to another more reliable hosting provider.

Update 19 Dec 2014: Neotel has still not answered the question regarding a “early termination penalty” due to our 2nd rack being commissioned in May 2014. This is puzzling to all of us, as we lit up both racks in January 2014 and there is no reason that one rack should have been commissioned 5 months later. We also experienced an outage on 9th December and it took Neotel 9(!) days to respond with a post-mortem (turns out that an aggregate core switch had to be rebooted as it hung) – despite asking about how this will be avoided in future and why there was no redundancy in place, those questions are never answered.

Update 29 Dec 2014: Another disaster at Neotel data-centre. Our bidorbuy infrastructure has been down since 11:30am due to the breakout of a fire (!!!!) in their UPS room. Although Neotel claims to be a tier-3 data-centre, the incident forced Neotel to shutdown power to the facility which resulted in all servers being brought down hard (SANs crashing, switches losing configurations etc). It is still puzzling, that despite being a tier-3 data-centre (where I would expect dual-power paths and redundancy) no redundancy was working and it was basically “lights out” from 11:30am till 4pm which is catastrophic for any online business. The first official notification we received from Neotel was at 12:53 (90 minutes after the incident occurred), the 2nd update was supposed to be at 2pm (received no update) and finally incident was closed at 3:52pm (although then their network engineers dropped our rack network configuration, knocking one rack off the network completely for another 2 hours).

Update 5 Jan 2015: Despite receiving assurance from a senior manager of the datacentre at Neotel on 29th December 2014 to receive an RCA (root cause analysis) within 24 hours about the fire in the UPS room of the data-centre which resulted in a 5 hour outage, we are still waiting for an official response. The account manager’s email account is bouncing due to it being full and most other staff is also not contactable due to holidays. One would think that a severe outage which resulted in widespread equipment failure and damage would receive more focus and attention.

Update 5 Feb 2015: In January 2015 we moved off Neotel and migrated to Internet Solutions. The account- and project-management with Internet Solutions was an refreshing and very professional experience (completely the opposite what we experienced with Neotel). The migration took one night (we temporarily switched to virtual hosted infrastructure provided by IS for free as part of the move) and another week to rebuild our physical infrastructure in the data centre. If you really want to be completely independent from service providers, there is one suggestion: Get your own Afrinic IP range.

Update 5 Mar 2015: One would have thought that Neotel is a thing of the past. But it’s not. We received a new set of invoices for April 2015 for racks provisioned in the Neotel datacentre (the same racks we decommissioned as part of our service termination). To be honest, billing is an absolute mess and we are still waiting for resolution of issues dating back to June 2014 and September 2014.