CASE STUDY
Customer
TekMasters provides global IT support services to the U.S. Department of State, Bureau of Consular Affairs.
Services and Scope
The services provided include, but are not limited to a variety of nonimmigrant and immigrant visa support services, such as processing information, setting up appointments, collecting fees and providing greeting services.
TekMasters provides the backbone of the IT Application Development, Infrastructure, Database, Voice, and Analysis support.
Mission Objectives
Provide business process support services for visa-related operations at U.S. embassies and consulates abroad.
Provide local, standalone facilities to offer a range of visa support services that either require and/or would benefit from the personal appearance of the applicant.
Provide skilled oversight and management of worldwide operations for all contract visa support services.
Provide customers with a positive experience by treating them professionally, efficiently, and with courtesy at all times.
Highlights
The program is run as a global e-commerce application that provides servicing to 53 distinct countries on a foreign national basis to include international currencies, courier services, and fee collections.
The application is responsible for over one billion dollars worth of fee collections per year.
The application processes approximately seven million applications per year with approximately four million of those actually granted.
The application processes 2000 requests per minute resulting in 2.9 million requests per day.
The project supports brick and mortar biometric collection in both the US, Australia, and 6 other distinct countries and is responsible for the operations, maintenance, and collection of all 54 countries tying into the DoS’s RDC system.
The project supports 24 ASC’s, Applicant Service Centers, in conjunction with DoS embassies where there is a high volume of processing needed.
Routes 13 million calls per year via VoIP and IVR solution.
Dynamic reporting and analytics on 2TB’s worth of data that is increasing by 2GBs per week.
Hybrid cloud solution that stores 1+Petabyte of data to be retrieved in real time.
Largest Consular Affairs Mobile Application that provides real time downloads on millions of user profiles.
Writing code is easy. It’s managing a project of this scale and magnitude that is the challenge.
TekMasters Program Manager
Application
TekMasters developed a robust enterprise application with the following capabilities and architecture:
TekMasters is responsible for gathering requirements, writing code, launching missions, and planning for the next mission all while managing tight deadlines. Agile SDLC was instrumental to this process.
Our application is responsible for live streaming of videos of all of our ASC’s.
Our application integrates with our IVR solution to process applicant payments as well as courier and credit card processing.
TekMasters has always taken a minimalistic approach to scaling with the ability to launch new features at any given time.
We have unified the approach for both immigrant and non-immigrant processing features.
We have harmonized for third party self services allowing ease of use.
Updating UI/UX flows.
Currently addressing portions of the applications in AWS for autoscaling.
Technologies utilized are:
RoR as main framework and programming language
JavaScript and AJAX for interactive and dynamic functions
MYSQL, POSTGRES, and NoSQL Databases for reporting
Capistrano for deployment
Chef for automated deployments
HAMEL for our templating language
Java for our biometric client application
IMPACT:
TekMasters had the ability to bring on new staff, with a behind schedule project and showcased the ability to ramp up and produce deliverables. Not only produce deliverables but deliver a solution with the ability to hold up to the initial load. TekMasters in the first year had 4 countries on line and delivered 30 additional countries in only 6 months. To accomplish this we had to stage rollouts involving huge countries with specific individual requirements that had to hold up to certain volume thresholds.
When TekMasters first inherited the project the web application was processing 200 requests a minute and currently is processing 2,000 requests per minute.
TekMasters has created the largest Consular Affairs Mobile Application that provides real time downloads on millions of user profiles that retrieves lists of applicants and supports the upload of check-ins for applicant processing.
VoIP and IVR
TekMasters built a customized solution using SIP based VoIP and Voxeo IVR to support multiple local and foreign call centers. Call centers provide the call center support to applicants who have questions on any part of the VISA application process.
The process of the applicant flow through the system is as follows:
Applicant calls a local number in country provided by TekMasters via a local provider.
Provider routes call to TekMasters hardware in DataCenter via SIP.
Rails application processes multiple variables to give the applicant options in their native language.
TekMasters routes and directs to the appropriate call center.
Call is tagged with a unique identifier to track and report which houses the applicants language, country, ref #, visa status.
Through the IVR system an applicant has the capability to process payments, schedule appointments, hear communication broadcasts, check on their appointments and view upcoming availability for scheduled slots.
Voxeo was chosen because of its intuitiveness, stability and reliability of not going down and also gives us redundancy from a VoIP perspective. It is also an open source utility allowing us more control over the infrastructure at a fraction of other COTS solutions.
We also use Splunk in conjunction with our IVR system for monitoring, reporting, and logging where we can monitor calls, traffic, audio, and QoS.
SIP/Voxeo/Postgres/CentOS.
IMPACT
With implementation of Voxeo IVR this eliminated the need to route to a Session Border Controller thus eliminating a hop, hardware leaks, ultimately providing a cost savings and giving us more control on routing calls via DNS.
Voxeo processes approximately 13 million calls per year and is responsible for 3 million dollars worth of phone payment processing.
Voxeo IVR has the ability to store specific recordings for all 25 of our language sets for the countries we support.
Database
TekMasters currently manages a 2TB MySQL database environment that houses all actions every applicant has taken as an audit log. Based off this information we provide over 50+ custom dynamic reports that go out to the field automatically. We are currently adding about 2GBs worth of data weekly over the 10-15 databases we manage.
Currently we utilize Jasper for our reporting solution.
The bulk of our data is stored in 2 big tables utilizing JSON as the best way to represent out data scheme. This is not the optimized solution which is why we are currently migrating our database over into MongoDB.
MySQL worked well in the beginning when we had small data sets of relational information that was formatted for easy look up.
MySQL/Postgres/MongoDB.
IMPACT
The move from relational to NoSQL will allow us to sit on a multi threaded shard and utilize Apache Spark for parallel processing. This will allow us to conform to guidelines as well as increase our speed and scalability for the data we house.
Reporting allows us to perform advanced analytics and view other database anomalies thus minimizing issues and improving continuity to ensure data integrity.
Network
TekMasters inherited a severely broken network that was comprised of Layer 2 switches with Layer 3 images on them. There were major issues with incorrect VLANs with segmented switches that were causing network traffic collisions. The network suffered from flooding, raised CPU levels , and routing loops which resulted in an unstable network with frequent crashes.
TekMasters performed a hardware refresh along with a complete datacenter migration.
Our network currently is comprised of 3 data centers, 24 ASC’s (applicant Service Centers) and 6 CIC’s (Canadian In processing Centers).
Built out a globally distributed private cloud network for Closed Circuit TV cameras (CCTV)
Implemented a centralized OpenStack storage environment to store High Definition CCTV recording from hundreds of cameras located in 22 separate facilities.
Storage consists of nearly a Petabyte of redundant storage and 10GigE networking.
6509e switches/dual power supplies/dual processors/cold standby/ASR 1002s/ASA 5585ssp20/OpenStack/XEN hypervisor/AWS
IMPACT
TekMasters ran all new cables, conduits and cable management all while producing a cost savings of 75% from the current set up.
TekMasters fine tuned network images resulting in a 99.9999 uptime in network reliability in support of a 24x7 global operation.