Why migrating to cloud based open-source database makes sense ?

Over the years businesses in the SME and Enterprise segment build applications and deployed prepackaged applications on multi-modal commercial RDBMS systems  such as My SQL sever, Oracle and DB2. These DBMS was first designed in the 70s when optimal utilisation of memory and compute resources was critical for performance of application transactions and query processing of various decision support systems.

Since then these RDBMS have evolved into a common IT staple for managing clusters of structured and unstructured data in business environment of any sector. In fact, they have a strong influence on how applications were designed and built in the past.

Most RDBMS vendors also built prepackaged applications that resided on their RDBMS systems further locking customers into their environment, though it must be noted that many of these applications were the foundation for business process automation and integrations that drove workforce productivity.

The complexities of running open-source databases in house..

Growing data and high cost

While many of these solution resulted in high performing systems of the time (pre cloud era) over the years businesses started to incur high cost to maintain the exponentially growing organisational data due to automation, mobility, web and sensors. These included DB licenses, ETL related tools and software licenses, storage, compute, and increasing number of human resources to administer performance of the data layer empowering various business processes.

Migration to open-source databases..

In much recent times, businesses have been experimenting with open-source databases for some applications where data growth is significant in order to cut database licensing cost. But this attempt resulted in various problems such as limited features (or features are not as robust as commercial systems), steep learning curve, expensive support and constant config, bug fixing, patching and performance tuning activities. While businesses were able to escape the licensing cost, maintaining a dedicated database team that keeps the open-source databases at an optimal level was a time consuming and arduous effort on the organisation's part.

Scaling based on forecast demand

To add salt to the pain, all of these cost are incurred based on forecasted demand as most businesses are scaling systems in a conventional environment (without cloud capabilities) which means cost is incurred way before the business hits the projected demand. Though this is a universal problem with most conventional IT architectures that lacks cloud capabilities to instantly provision, scale, update, patch and de-provision based on internal demand.

New cloud based applications and microservices...

As we enter the cloud era, application architectures started to take effect of the SOA capabilities of cloud, with high levels of abstraction in every layer (e.g infrastructure, data, intergration and application).

A polyglot architecture where developers can take advantage of the best DBMS, programming language and other ready cloud services including AI, IOT, and Bigdata that accelerates application delivery, begin to take center stage. Apart from SQL type of DBMS, no SQL requirements to support other data types such as value store, document, graph, time series and in-memory DBMS started to emerge as demand rises to 40%. All of a sudden, purpose built DBMS were starting to look much appealing to developer communities every where as opposed to multi models.

Choosing to build such an applications on commercial database can make projects less viable in terms of both cost and feature sets required for a cloud environment where service orientation and scaling instantaneously based on demand is of high importance.

Why cloud based open-source databases are a good option?

Thanks to increasing popularity of public clouds, open-source databases today are mature enough in terms of features and support for wider adoption in businesses including large enterprises.

Overcome management constraints posed by on premise deployments...

It's common to find MySQL, PosgreSQL, MariaDB, MongoDB and countless others in organisations' on premise environment, underpinning many critical business application both prepackaged and customised, but then those implementations are plagued with similar issues surrounding commercial databases such as scalability, availability, security and high operational cost to keep it updated and running.

Cloud providers' DBPaas services eliminates this issue by removing day to day operational complexities from businesses that just want to use the services to build various business solutions to increase quality of engagement with existing and new types of audiences.

Wider adoption and value extension is now possible....

With database administrative task out of the equation, tech teams can now focus on migrating more applications on open-source databases for better value and standardisations within the organisations.

High priority business requirements can be addressed much faster than before with available team and funds, curbing the need for any shadow IT by LOBs.

Refactor apps for the cloud..

Another indirect benefit of this exercise is to modernise legacy applications sitting on premise environments. The migration process is an excellent opportunity to learn more on what pure cloud based databases can do for some applications.

Many cloud databases such as Aurora, Dynamodb, Azure, Google Cloud Bigtable, Google Cloud Datastore, Aspara and DocumentDB are quickly maturing in feature sets and support for other open-source database engines especially MySQL, PostgreSQL, MariaDB and etc. These cloud databases are designed for cloud applications and is highly service oriented to scale unprecedentedly in a highly available and secure manner.

Taking advantage to create low risk initiatives to migrate at least some applications to these cloud databases can result in elimination of years of technical debt inflicting your IT environment.

So, where to get started ?

Identify the open-source equivalent of your commercial database/s

Best method is to start from DBMS systems that contributes to a significant share of the licensing cost and prioritise those pending upgrades and require additional licensing. Identify how many applications are affected, documentations, customisions and integrations to external systems.

Research potential open-source databases and their fit to replace your current commercial databases. Upon a selection, start segregating apps that can and cannot be migrated to the new target database.

Identify cloud based open source database providers....

Next research and identify cloud vendors who provide DBpaas services for open source and cloud based databases. Most cloud vendors such as Aws, GCP, Ali, IBM, Azure provide services for MySQL, PostgreSQL, Maria DB, MongoDB, Firestore and other popular open-source databases. They manage bug fixes, patches, upgrades and scaling, along with day to day operations enabling a far less number of DBAs to manage a bigger number of database clusters.


Rate migration complexities for each app...

Once service providers of choice are identified, explore portfolio of tools provided and recommended by user communities (3rd party) to make an initial assessment on migration complexities of the selected applications. When choosing or building migration tools, think of repeatable usage for both data and refactoring requirements.

Once the potential tools are selected run some initial rest to rate apps, based on migration complexities (single step, multiple step, data size, refactoring levels) and other criterias such as critical app, number of users and etc. At the end of the phase select your pilot case or cases.

Contact 1 or 2 cloud providers to start experimenting...

Next contact your current or suitable cloud providers to compare services and pick one that best suit your business in the long run. You can decide this based on range of service offerings, supported open-source DBMS and cloud databases, price-service effictiveness, cloud eco system offered (other services, partners, user communities, availability zones and etc).

Upon selection of vendor, start experimenting your pilot cases ranging from low to high complexity. Document challenges faced and resolved with future migrations in mind, which means processes must be repeatable and automated as much as possible.

Create a migration schedule and milestone for large scale impact

Having assessed pros and cons of migrating to open-source databases operations to the cloud, build a schedule for migrating apps to the cloud and start executing them in line with business objectives. Leave room for changes in the event new priorities enters the business.

Start large scale migration of all target environment..

Upon completion of pilot migration case, refining and further automation of tools start to migrate applications with low complexities first to the new cloud based open-source database, one at a time.

Address any refactoring needs and test each service before moving to the next, till all low complexity migrations are completed. Repeat the process for complex migration projects. Keep updating schedules, findings and refine tools if neccesary.

Relieve resources for high impact projects

At the end of this process, your team would have not only realised licensing reduction and much stable performance of your databases powering critical applications but eliminated at least some technical debt around critical applications and relieved valuable IT resources for other high impact projects to build new business capabilities, improve access to business insight and performance of queries instead of maintaining 'lights on' projects.

Navigating Values - Cost, Technology Debt, Data ROI and IT Resources

Often a open-source database adoption is initiated due to lack of funds for commercial databases or to cut back on commercial licenses that results in millions of dollars of recurring cost.

Cost

But moving to a cloud provider and consuming DBpaas services results in cost reduction in all types of licensing including those used in ETL processes. In addition , users need not incur cost until a demand arises. This changed how tech organisations plan for future needs, while allocations are made, they don't have to be spent until need arise.

Technology debt that hinders availability, scalability and security design

Another key value is the migration process trigger application modernisation activities to refactor key business solutions on premise to scale and perform better in the cloud both public and private. Migrating to cloud based databases such as Aurora and Azure increases these values by leveraging services enabling high availability, concurrency, business continuity, security, and scallibility.

IT Resources

These values are best navigated with scale of adoption horizontally and vertically. The wider the adoption the higher the value accross the organisation, freeing resources time, money and people to do what matters most to your growth - improving experience for customers, creating business innovation, increasing efficiencies and speed to react to market opportunities.

AI Playbook for Startups and SMB

The AI Playbook for SMBs and Startups

Most businesses in the SMB and startup class, still struggle to grasp the broad definition of AI disciplines, what AI really is to their business, where to get started and what successful execution looks like for their businesses. Simplifying a focus area narrowing these concerns specifically to address your business pressures is helpful to keep team in line with long term business goals and strategies.

There are many AI Playbooks written by renown gurus such as Andrew Ng, meant for Fortune 500 organisations and they do contain some very useful guidance and tips for all businesses. Nevertheless, smaller ventures with limited funding, skills and resources may need to rework tactics to suit business.

Here we present steps to formulate a generic AI playbook meant for small and medium sized businesses, which includes stages to:

• internalise AI definition for the business; 
• outline and prioritise business problems that can be addressed with AI;  
• experiment with ready pre-trained model
• build required skills or internal structures;  
• prepare training and test data; 
• choosing a path between pre trained and custom built model for solutions, 
• and assessing overall impact to businesses performance. 

The generic AI Playbook should ideally incorporate other dimensions unique to your business, industry and commercial eco-system over time, for maximum results.

Define and internalise AI for your business...

AI itself is a broad subfield of computer science. It encompasses many branches of studies including rule based AI and decision trees; machine learning (regression, classification, neural networks/ deep learning, reinforcement learning, supervised and unsupervised learning); robotics; text and speech processing (NLP, NLU, NLG); vision (computer vision, image recognition) and several other applications that still have not left the laboratories. 

AI is a field where one or several of these approaches are used to create machines, agents and models that perceive, learn and adapt to perform various tasks and cognitive function similar to humans.  

The AI domain is practically exploding with hundreds and thousands of new approaches, methods and solution areas in transfer learning (learning from lesser data and smaller models), networks with memory (generalising AI, e.g. DeepMind’s differentiable neural computer), reinforcement learning, training simulations, hardware for training and inference that are rapidly advancing in the research, development and real world application stages. 

Communicate how AI should be perceived in your business....

As such, it's easy for businesses to get lost in an ocean of offerings that comes with vendor specific methods, best practices, strategies, tools and ready use cases. Charting and communicating how AI should be perceived within the business very early on, in your AI journey is crucial to ensure potent utility, informed expectations and performance. 

For instance should AI merely assist workers to perform better at tasks? Should AI considered as an avenue to outsource cognitive functions otherwise performed by human workers ? Or should AI be a form of cognitive technology to help expand ordinary worker's ability to explore meaningful solutions to business problems both creative and logical?

Setting the right internal tone on AI can guide alignment with work culture, strategies, high impact projects, investments, selection of tools and services that can drive AI values across the business.

Define business problems that can be solved with AI

The common misconception in this stage is, data science or business team alone is responsible to outline business problems solvable by AI. 

Every business unit can contribute to this stage including sales, marketing, services and tech departments. Imagine how it will affect your business if tech can reduce 40% of cooling needs in your datacenter or how your customer service or inbound sales department can improve performance by multi-fold with realtime speech to text analysis that prompts them with emotion, sentiment and other useful information during a call with a customer.

Build working team; conduct regular meet-ups; answer business questions around growth hacking, improving or staging customer experience, workforce productivity and innovation; keep a revisable record of issues pressing the business growth (this can be a simple excel sheet with list of items); prioritise problems to solve; assess data availability; experiment, assess skills requirement; select initiatives for executive sponsorship and create projects to execute. 

Assess data availability and prepare training data for the problem

Data sources...

Data can be gathered from within the business, public datasets, procured from external sources (e.g. Image Net, Google Data search) or a mix of these. Identify useful data assets for your business problem, list what data is missing or not available and which data can be excluded from your raw selections. Collect and aggregate sufficiently to fit your model. 

Data preparation is a significant contributor to cost of building AI solutions specifically ML models, aside from high performance computing resources though GPU innovations has brought so much economics since a decade ago.

Clean and transform data for training...

Most ML models requires large amounts of data to train and converge before they can be applied to business. Preparing these data may require domain experts to collect raw data, identify features with strongest prediction power and label them. Though the business problem you are aiming to solve, tools and selected algorithms ultimately influences data preparation approach and mechanisms. 

Data transformation is revisited several times....

It is common for developers to spend 50% or more time preparing and scrutinising data quality and revisit data transformation processes (or feature engineering) several times during the development of the model. 

Split data sets for training, test and validation...

The prepared data is usually split into training set to fit the model, test set to evaluates final model fit on the training dataset and validation set to evaluates model fit on the training dataset while tuning model parameters.

Consider pre-train models

Ready to train models for various use cases...

For common business problems which requires simple recommendations, classifications, regression, clustering, anomaly detection and ranking, look for pre-trained model offerings by public cloud vendors such as Google, AWS, Alibaba and Microsoft. Many large and niche AI vendors now offer pre trained models for other slightly complex solutions to perform predictions, speech processing, computer vision (video, image and object recognition), decision support and planning tasks. 

No need to build algorithm or ML models.....

These offerings saves businesses the hassle of building algorithms and models from scratch. Plus it's a great way to get started with applied AI for business with quicker results and fast ROI times without having to reinvent the entire process (the value is in the model outcome not building it, especially for non-tech businesses). 

No need to replicate work for  ML backend .....

The team need only concentrate on preparing data, select features and train the model for required tasks (e.g predictions, classifications).  Trained models/ functions can then be deployed with other production applications and systems through APIs provided by cloud vendors to apply learnings on new data samples without replicating any work for ML backend.

ML and DL frameworks

In the event that the business problem you aiming to solve is much complex and does not have a ready pre trained model available, it is advisable to custom build the model custom build the model on a machine learning and deep learning frameworknof your choice, offered by cloud vendors through their ML-as-a-service offerings.

Frameworks such as Tensorflow, Pytorch, MXNET, Keras, Caffe, Scikit, Microsoft Cognitive Toolkit and Theano, comes readily packaged with interface, libraries, tools, pre-built and optimised components to facilitates fast development of AI models without getting into the details of underlying algorithms and complex architectures. 

New developers can benefit from starting with frameworks such as Keras and delve deeper into Tensorflow or other suitable frameworks as they get accustomed with the building blocks and programming languages.

However, it is highly recommended for new teams to get experience from simpler projects involving pre trained models first before venturing into challenging custom building models. It is easy to loose ROI when tinkering with ML or DL models without past experience and benchmarks.

Building AI skills 

When it comes to smaller businesses, building AI skills in-house is really a trial and error exercise to find the right blend of subject matter experts (or the minimal quo) to address AI problems and build solutions. This team's skill level is adjusted according to how they plan to address business problems, for instance with ready models or custom build models.

AI requires programming (e.g. R, Python, Lisp, Prolog, Scala, C, C++ and Java) data science (at least fundamental level) and business domain knowledge at the minimum to obtain some high level results. A strong foundation in mathematical fields such as probability, statistics, linear algebra, mathematical optimisation is necessary if your business wish to develop own algorithms or modify existing ones to fit specific goals and constraints.

Training existing workforce, hiring from other organisations and university grads are some of the common methods to build the right team.

Education sites such as Coursera, EDX, Machine Learning Mastery and many others provide neutral self learning content suitable for both technical and business audience. Content varies from fundamental to advanced level and comes with certification process that can be committed at one's own pace. 

Vendors such as AWS, GCP and Microsoft too have their own academies but syllabus may be skewed towards respective services. AWS and Microsoft are so far the best corporate academies I encountered and complies to many criteria mentioned in Andrew Ng's Fortune 500 AI Playbook in addition to easy to follow content and no fuss lab use (note that this is paid resources).

Assessing impact to business..

Once trained, tested and deployed, weigh the model's impact to your business performance. I use a no fuss method where I categorise what was increased, reduced, created and eliminated over a set period of time.

For instance:

• Did the model INCREASE overall customers value, revenue or profitability ? 
• Did the model REDUCE fraud risk, customer churn or cost ?
• Did the model CREATE new growth engines, data products or differentiators ? 
• Did the model ELIMINATE manual task, blind spots or redundancies ?

These answers are then scored and quantified for best alignment with overall strategic goals and objectives. The data is then collated for all AI projects to visualise and quantify the total impact to business financially (profit, revenue, growth rates, etc) and other areas such as competitiveness, reputation, and customer satisfaction. 

This exercise is helpful in two or three ways :

• to identify successful AI projects and approaches
• to build AI projects that can increase the value or performance of existing models
• find optimal opportunities to apply AI learnings to other functional areas 

There are many ways to measure impact, choose one that you are most comfortable with and put it to work. Use the discoveries to increase effectiveness of AI projects for your business while filling gaps in skill sets and access to useful data assets.

Conclusion - Ethics, Explainability and Human-Machine Interaction

In the end, AI systems are nothing but stacks of infrastructure and programs in various architectures that learns and executes continuously, depending on its utility. The machines aid human workers to predict outcomes, navigate a situation and decide on best action path with less mental drudgery while spanning solution options that was never possible before.

But that doesn't mean AI systems and models are perfect or free from self vested exploitation. Since the 2010s public concerns about racial and other biases in the use of AI for criminal sentencing decisions, creditworthiness and other forms of social credit systems (e.g. China) is driving demand for transparent AI systems and agents.

Explaining decisions of AI algorithms or models, especially those powered by deep learning or other complex neural networks is still a difficult task and is also known as the 'interpretability' problem.  Unlike decision trees and Bayesian networks that are more transparent to inspection, complex neural networks stack on one another and may involve millions of neurones processing towards the outcome.

As such, deploying sufficient governance protocols for data privacy and security; an architecture which facilitates transparency and tools to detect bias in algorithms is as critical as creating successful models or functions. Though in many cases engineering for more explainability and transparency may lead to significant accuracy trade offs. 

Making Sense of Public Cloud Investments

The public cloud infrastructure services (Iaas) market, according to Gartner's estimation is reaching $60 billion in 2019 and is growing at 27% YoY (fastest growing cloud services segment) from the previous year. Assuming Asia contributes 25% of the global share, turns the region into a critical geo for cloud vendors to battle for market leadership. 

However, cloud practitioners in Asia are still somewhat battling with what is considered as decade old cloud myths evolving around security, compliance, IT control and short term ROIs when conversing with regional customers riding on conventional structures and work cultures. This includes financial institutions, communication service providers, retailers, manufacturers and other asset intensive industries.

While these are all valid concerns, over the years premium cloud vendors have more than proven that their infrastructure and platform services (IaaS, PaaS) are superior to any single institutional  environment in terms of performance, security, reliability and overall economics to sustain just about any type of workloads. The public cloud values are now evolving rapidly with introduction of newer, advanced services in opensource databases, artificial intelligence, low code coding, and running mission critical workloads, enabling businesses to gain better control of operations as they cut waste, optimise performance, develop the much needed new capabilities and differentiators faster than ever. 

Open-source Database Services

Businesses have been dabbling with open-source databases (as well as open-source operating systems, libraries and tools) even before cloud computing services emerged and was aware of the potential to free organisations from growing commercial database licensing burdens. However, managing these open-source databases required additional training, operational and maintenance resources, apart from the unpredictable fixes and releases from the open source community. These issues limited the usage of open-source databases such as PosgreSQL to a small number of non critical, in-house development projects and applications in the past. 

This scenario changed in the recent years as cloud computing companies started to offer fully managed open-source database services such as PostgreSQL, MariaDB, MySQL and many others. Businesses start to increase adoption overnight by consuming cloud based open-source database services for existing and new applications while gaining new efficiencies, never possible before. The polyglot nature of cloud native development, where multiple types of databases and models are deployed in an application composed of micro-services and API calls only drives this trend further. 

Popularity of open-source databases according to DB Engine Ranking site, is about to converge with commercial databases in 2019 or in the early parts 2020. 

Artificial Intelligence, Machine Learning, Deep Learning

Explore pre-trained ML models

If new to artificial intelligence and its sub domains machine learning, deep learning and transfer learning, take advantage of the various pre-trained models offered by public cloud vendors such as Google, AWS and Microsoft to perform prediction, classifications, natural language processing (NLP), speech processing, computer vision (video, image and object recognition), decision support and planning tasks. 

Start by experimenting solutions to business problems while inflating returns on organisational data assets. This provides an opportunity for the business to quickly reap benefits from readily available models where the team need only prepare data, select features and train the model for required tasks (e.g predictions, classifications).  Trained models can then be deployed with other production applications and systems through APIs to apply learnings on new data samples.

Building custom models..

As the internal team gain experience and get familiarise with ML projects, commence a process to create step-by-step AI playbook to define and prioritise business problems that can be addressed with AI, data preparation, algorithms selection, and developing a model to make predictions. This exercise should involve critical members from business, data science and the engineering teams, ideally aligned to addressing growth challenges, resources optimisation, innovation and enhancing customer experience. 

Machine learning and deep learning frameworks such as Tensorflow, Pytorch, MXNET, Keras, Caffe, Scikit, Microsoft Cognitive Toolkit and Theano, that comes readily packaged with interface, libraries, tools, pre-built and optimised components facilitates fast development of AI models without getting into the details of underlying algorithms and complex architectures. 

Pick a cloud vendor for ML-as-a-service that best fit your needs...

Top public cloud vendors such as AWS, Microsoft Azure, Alibaba and Google provide support to most of the frameworks mentioned above for custom building models apart from their pre trained model offers for common business problems. New developers can benefit from starting with frameworks such as Keras and delve deeper into Tensorflow or other suitable frameworks as they get accustomed with the building blocks and programming languages.

Develop cloud native applications faster ...

Today a challenge sparks an idea, an idea turns into a prototype and a prototype transforms into a multi-platform first release in just matter of days and weeks. Aside from core infrastructure services, cloud vendors offer various advanced services ranging from devop, language SDKs, container orchestration, serverless computing, no to low code development platforms (both proprietary and open-source), APIs, code libraries to ready templates to help organisations accelerate application development arising from sudden business needs. 

Most post cloud applications are series of micro-services connected to one another via APIs leveraging languages, databases and code bases most suitable for the required function and the supporting technical environment.

No to Low Code Platforms

In addition, low code platforms such as Mendix, Outsystems, Google App Maker, Appian, and Microsoft Power App empowers traditional developers, IT professionals and to a certain degree the average business users to accelerate software delivery for business use cases made up of common features and components by simply utilising existing templates, forms, objects, drag and drop of prebuilt features.

Ensure performance of mission critical applications....

Learn which of your mission critical applications are suffering from intermittent or persistent performance issues affecting business processes critical to productivity, customer experience and revenue activities. Cloud vendors offer various services and pricing plans to optimise business applications from SAP, Oracle, Microsoft, IBM and others where data grows exponentially with transactions in time.

Exploring public cloud offers for the aforementioned type of applications can reveal credible areas of savings, operational simplification, performance improvement, speedy delivery of new business requirements, higher utilisation, improved security and compliance. Keeping a revisable record of requirements and working with cloud vendors to formulate cloud contracts to efficiently counter all planned and unplanned workloads can free internal resources for myriad of higher impact business activities.

Know values, learn constraints and strategise for long term...

Ultimately cloud values can vary significantly from business to business depending on the nature of operational model and the overall ecosystem. The four areas mentioned here are general enough for most sectors, though some businesses might benefit immensely from other emerging tech such as augmented reality, IOT, 3D or bigdata platform services.

In addition, identifying constraints that prohibit enterprise architectures from extracting full potential of public cloud eliminates aimless unproductive explorations. For instance legacy apps on rigid centralised architectures that may not gain much efficiencies when moved to the cloud.

Finally having a solid long term strategy, helps. For instance all new development and deployment to take the 'Cloud First' approach, or maintaining a hybrid cloud environment with only unpredictable workloads moving to public cloud or 'Cloud Only' approach where the internal teams build in-house brokerage capabilities to fully monitor its environments on multiple public clouds and perform various optimisation as needed.

LK Expert Opinion featuring Peter Adamson – The Mining Sector, People and impact of Technology Advancements

It was sheer privilege to conduct this interview with Peter Adamson, an accomplished general manager, consultant, advisor and people advocate in the mining field with broad global experience in business, organisational effectiveness, operational readiness and safety culture. He held various leadership roles in the mining industry, driving multi billion dollar operations across continents, including the Americas, Africa, Europe, Asia and Australasia.

In this casual interview with LK, he shares his words of wisdom, global experience in shaping leaderships and witty responses on how he thinks technological innovations are changing the sector, decision making processes, environment and future workforce.

Find the full transcript of the interview below:

1. Tell us a bit about yourself and how you got started in the mining industry?

What a hard question!  I ended up here totally by accident, perhaps because of a background in community development and people issues, gravitating to an industry where the main asset is not the machinery or even the ore bodies. The main asset in the mining industry is people who come together to run the businesses.

My first involvement was conducting rescue team training, which seemed to quickly become team development and they were the first two aspects of my work in this industry.  Of course, team development is also about leadership.

My passion for learning has always been about the power of experiential learning and that’s where I became engaged with this very interesting and enormously diverse industry. Somehow, building managerial leadership skills in others has meant that I have ended up in senior General Management roles myself, when I have not been consulting (Learning by doing).

2. The mining industry is seen by most as having adverse effects to our environment, work safety and social fabric (especially native communities), both during and after mining processes. What are your comments on this?

The quandary for me is that, there are so few human activities that are not harmful or adverse to our natural environment.  Geologists and miners are living and working in natural environments, this is our backyard, and we do (in Australia) demonstrate a high level of care for that backyard.  Perhaps, more than any other industry in the world, mining seems to be at the forefront of investing in knowledge, skills, technologies and the means to mitigate our impact. Compared to farming, garments and the travel industry, mining is doing quite well if it is about comparison, and what can be done, particularly so in Australia. Everything that we ‘people’ do is harmful to the natural environment.

Regarding safety, yes, when I first came to this Industry, it was accepted that it was of high risk, that people would be killed and hurt.  Over the years, we have been able to change that belief to one where, no one needs to get killed or hurt going about their daily work.  Statistically, in organised mining in the Western World, the most dangerous part of the workers day is driving to work.  Fishing, (particularly recreational fishing), farming, and other activities are extremely dangerous now, compared to mining.

Across the past thirty years, one person in one team, did cut his thumb, requiring a suture, driving a stake into the ground without wearing his gloves. This is the only event, across tens of thousands and thousands of people days working in very hazardous environments doing risky work in different parts of the globe.  This is aside from illnesses which are still problematic in many parts of the world.

Regarding social fabric.  I have worked with this element in many countries around the world.  Mining is an industry which creates wealth and in so doing can easily generate poverty as the projects go through their various stages of development.  If managed well, and with thinking going out beyond the intergenerational change, the disruptive opportunities for positive change that mining can generate are enormous.

Think health, education, training, economic development, information flows, small and large business development, technological development, research in a wide number of fields, new skills, useful employment and empowerment of people.  The variations on this are enormous and most of it, is quite healthy if done well.  When it is not done well, poorly thought through, working from flawed principles, ad hoc or where a vested power seeks to maintain the status quo, then it can mean horror.

3. What action would you take to change this public perception?

Each time I have been involved in a project, agreement of the key principles for how the project will be conducted, is where I start.  If I cannot accord with those principles, I walk away.  Transparency, Zero Tolerance for Corruption, Integrity, Safety and Compassion are some of the key drivers for me and for successful Mining Projects.  Strange words to associate with such an apparently harsh and exploitative industry but these are key. When we get it right, no one will notice, is how it should be.

4. How does technological advances impacting your industry?

Enormously. There have been no new technologies now for some fifty to sixty years, but our abilities to advance existing tech, build and create from previous inventions, still seems to be in early stages.  Imagine a future where we tip a cup full of nano particles onto the ground, they soak down and organise themselves around an orebody, explode or expand, then transport the ore to the surface with no people involved.  In the same way, we may go after cancer cells in the human body.

The significant downside of technological changes are in not knowing how best to use them in our industry, and the impacts they may have on the purposefulness of the humans in the industry. Reducing employment numbers meets cost objectives of the miner, however the impact for employees is significant. We can now, do so much more, with so much less and at the same time reduce the opportunities for harm.

5. Most people associate this industry with adventures, explorations and dangerous jungle expeditions in remote lands lead by the instinct of daredevils alone. How data dependent is this multi-billion business in reality, especially during prospecting or when operating a mine?


I have had some wonderful experiences with some awesome people in all sorts of places, around the world. It has been great. But now, we are being tamed!  Yes, there is a place for experience and intuition in this industry, however it is diminishing. Data, data and data are the critical elements. Appropriate technology for generating information that captures all of the data is key.  From that information, choosing which bits to use for forming the organisational knowledge bank, that’s where the intuits are still having some say. The growing science to deal with this in a useful way will mean, soon the term “knowledge Management” will take on new contemporary meanings, I am sure.

6. Are technologies such as mobility, IOT, AI, Augmented Reality and cloud computing a common tool for collecting, sharing and analysing data today in this industry? How do mines connect with their corporate offices? What business events can you predict?

The data generated in mining operations is of a magnitude very difficult to comprehend or describe, it is on a scale so enormous.  Engaging with that data and accessing it to create knowledge and generate decisions for ongoing monitoring and assessment, whilst working collaboratively with fellow professionals.

I have sat in an office in West Africa with real time data linkages into Toronto, London, Paris, Perth and Melbourne whilst my consultants have had similar linkages back to Delhi, Houston, Sydney and other places. One project with an 18 billion dollar capital expenditure meant we were spending 48 US Million dollars a day, the magnitude of just tracking monitoring and ensuring this happened in the way we wanted and the technology we used, was probably at leading edge.

With the wave of autonomous vehicles and long distance trains, this is the realisation of a vision from twenty five years ago and the focussed development of appropriate technologies along with synergising opportunities.  It varies from trucks and trains repetitively in pattern, to machines deep underground working autonomously, all controlled and maintained by an operator thousands of kilometres away.

For example; can you imagine the data, knowledge and systems thinking that goes into deciding to drill one oil well, when the cost of that well is likely to be in excess of $70 million USD?  When the measures of risk and uncertainty for the engineers are so vastly different for the same measures of risk and uncertainty for the geologists, these professionals live in such different worlds but need to reach agreement.  Gathering evidence for and against that agreement and assessing the evidence that is missing, that is one of the key roles of technology in that setting.

7. Some people say that some of our landfills contain more metal concentration today as opposed to the mines. Is this true?

Who knows?  The metals we are mining from the earth are a finite resource.  They do run out. South Africa has passed peak gold, Australia is probably the same. Other minerals too, Australia long ago passed the peak of its diamond production, although with all of these, new resources are waiting to be found.

8. Do you think we have a mature enough recycling technology to recycle metal concentration in the landfills?

Landfill has to be a resource. We did not do the planning with landfill, with the idea to make it accessible again, and much of what is down there now, is probably absolute waste.

Changing our lifestyles will probably generate greater opportunities to reduce the impacts on our world.  Just deciding to take the colour of our nail polish, motor car duco, and other colours available in paint would have a huge impact on mineral earths mining industry.

9. Do you think that technology advances have also resulted in growing demand for certain rare minerals?

Definitely, and it has generated some dodgy sovereign practices too, where countries have attempted to corner a market so as to have global dominance, whilst other countries have sought to erode that dominance. Rare earths is one example. Some of the strategies by different countries to assert themselves globally through minerals are interesting to behold, they go back a long way in history and continue to provide an interesting spectacle, although activities are usually much more covert in this past century.

10. How do you envision, the future workforce of this industry ? What kind of new skills and expertise are essential to embrace technological advances that will further optimise business processes? Do you think that the mining workforce will consist of more people with multidisciplinary qualifications including computing and data expertise (both in the field and corporate)?

It would be hard to see even more computing specialists in the industry, than the thousands already here.  More I see a broadening of skills bases and a change in stratum thinking so that we all have the computer skills, plus there are still specialisations but the changes should be measured in a different way.

For example, in the Australian Mining Industry today there are few jobs where people are just working “in the day”.  And those jobs are rapidly disappearing.  The minimum that people will be thinking and working in, will be across 90 days result areas. As this thinking moves out from the mining industry into other sectors, I see broader changes happening globally.

Truck drivers, cleaners, bulldozer operators and train drivers are all now being replaced in this industry with broader ramifications for other industries.  We no longer need to have the farmer sit on their tractor whilst ploughing, seeding or harvesting their crops. Those were the jobs where people worked in the day, most of them disappear in the near future, I am not prescient enough to know what fills those spaces.

11. In your opinion, will we ever reach to a point that issues such as ‘conflict resources’, is a thing of the past? 

If only, I knew the answer to that one, the dream of world peace and the disappearance of poverty and greed. A world of compassion and empathy.



LK's Note : We hope this post has been both educational and inspiring to readers. If you have an interesting question for Mr Adamson, kindly post them here.

Application Portability when Switching Clouds

Every business develops and runs a fair bit of custom code applications to automate value chain delivery to customers, partners, employees, and even supplier interactions. Highly progressive businesses are driven by data which means these businesses continuously build, integrate, stage and deploy custom code to improve a range of outcomes namely process automation, user experience, operational visibility, compliance, and governance at dizzying speed.

Cloud services both IaaS and PaaS are natural choices today for development teams when it comes to realising business services internal and external. In fact, Rightscale recently cited in their ‘State of the Cloud Report’, that in average 48% of SMB workloads run on public clouds.

While healthy public cloud adoption is projected to continue among SMBs at slightly higher pace than enterprises, there is no denying that decision loops concerning the tools and cloud services selected are primarily dependent on the development models (continuous integration / continuous delivery) in practice and often comes with some degree of compromise made between available budget, source code control, speed of development, application portability and interoperability in the future.

Requirements and priorities change

Business priorities change as product or service adoption increases, market factors open new opportunities, competitive landscape tightens or new innovations enter the playing field, all of which involves scaling at a pace never imagined in the pre-internet era. Public clouds large and small, without a doubt power millions of businesses to build solutions quickly and cost effectively with a lot less overhead to meet growth objectives. However, a judiciously selected cloud partner today may cease to be relevant in a future point.

Over time, businesses may need to reassess relationship with existing cloud providers for various reasons ranging from  cost optimisation, security, governance, vendor lock-in, overall performance, cloud bursting to new technology requirement for AI, IOT, VR/AR/MR, 3D and mobile, making application portability a key concern for businesses wishing to switch cloud providers.

Switching public cloud providers in alignment with business goals

Many ISVs for instance may have chosen to develop and deploy new age applications such as IOT for instance, on a certain PaaS service to reduce cost of building the infrastructure and platform themselves. As business grows, the once economically viable solution begins to blow out of proportion financially as usage base explodes. In this case the business is left with two scenarios; one they could look for another provider with similar or equivalent PaaS services at better price points; the second is to choose a IaaS provider and construct their own PaaS layer for more control over application dependencies (e.g. data services, runtime, OS, networking, encryption, and virtualisation).

Application re-platforming processes and routines differs cloud to cloud

The cloud platform on which an application runs and DevOp environment determine how tools (e.g. Ansible, Chef, Puppet and etc.), automation and standardisations are applied in simplifying deployment when migrating applications from one cloud provider to another.

While a standard process perhaps can be established to automate packaging, conduct optimisation, generate VM or container images and orchestrate deployment artefacts, the knowledge sets or expertise involved varies from one cloud platform to another. Though, there are several steps and best practices that can be adopted by DevOp to ensure portability of applications and data today.

Choose cloud providers that support Open Virtualisation Format

Chose to work with cloud service providers that support widely accepted application packaging formats such as the ‘Open Virtualisation Format (OVF)’, formulated and recommended by the Distributed Management Task Force (DMTF). This method standardises packaging of virtual machine based software regardless of cloud platforms which means migrating to a new environment or multiple clouds is swift and resource efficient.

Containerise applications

Another best practice that is gaining momentum recently, is to re-platform applications running on virtual machine instances to a container based platform such as Docker. Containerisation packs application code and all dependencies in a container image that is a repeatable artefact and multiple containers can be deployed on a single VM sharing the same resources. Aside, from the obvious administrative efficiencies and cost savings, this method is ideal to support microservices development architecture (or polyglot architecture) and distributed applications on serverless cloud instances and virtual machines.

Container orchestration and cluster management solutions such as Kubernetes and Docker Swarm facilitates orchestration, clustering, scheduling and integration of large number of container images and typically is sufficient to scale applications as needed. Consider both tools and a proof of concept with each technology, using real-world workloads.

Open standard interfaces and APIs

Open APIs and vendor specific APIs (extensions of SOA architectures) allow users to move data and expose services between clouds while masking complexities behind any one interface (e.g. functional, admin, business). In a real world situation the use of vendor specific APIs are unavoidable even in a ‘open standards friendly’ DevOp environment, especially when the business footprint in a particular cloud is significant.

API management is mandatory in maintaining balance and isolating refitting work when necessary. Security concerns should be maintained throughout the linkages by deploying third party access rights management and other tools, as data securely travels between clouds.

Adopt multi-cloud management platform

Another method to facilitate cloud migration is to implement a suitable multi-cloud management platform that integrates multiple public cloud environments (e.g. AWS, GCP, Azure, Digital Ocean and etc.) including private clouds. This creates an overlay that masks underlying platform differences that allows for easier movement of applications, workflows, and data.

Build own PaaS

Vendor lock-in and portability issues are higher in PaaS services than in IaaS. One option to reduce such reliance is to build an in-house PaaS with tools such as Dokku and Flynn, an open source platform solution for everything that is needed for running applications in production.

Conclusion

Portability and interoperability between clouds is a journey that begins from the point a business on-boards its first cloud service.

Containerising applications, automating deployment processes across multiple clouds, building tools, creating a controllable PaaS layer or at least selecting one that has an equivalent alternative, leveraging open technologies wherever possible, using standard APIs and abstracting integration layers are some of the best practices that can be adopted throughout a cloud lifecycle to ensure smooth exit when need arises. These same best practices reduces vendor lock-in,  unexpected cost spirals, time and material needed to move workloads across public clouds.

Applications and systems with below average portability scores simply don't make the cut as a good migration target to other clouds. In some cases, it might be even easier to just recreate them on a new platform.

Migration processes too must be planned and executed in systematic manner with necessary tooling and automations in place to create a repetitive migration asset that includes the learnings that can be shared with the entire team.

Last but not the least, when porting over to the new cloud, systems must be fully tested and stabilised prior to terminating the old environment in ensuring business continuity.