Harnessing big data for a smoother maritime journey

18 January 2024

In the vast expanse of the maritime industry, data isn't just big; it's massive. However, big data isn’t simply about volume. According to a research paper by the World Maritime University (2021), big data describes large volumes of high velocity, variable data that require advanced techniques and technologies to enable the capture, storage, distribution, management and analysis of the information.

Big data on the high seas

Traditionally, the maritime sector has drawn insights from sources like ship logs and registration records. However, with the rise of new technology, such as sensors, satellite-based receivers, Artificial Intelligence (AI), and the internet of things (IoT), ship management is now heavily reliant on real-time data from a vast array of sources. As a result, big data has become big business. This wealth of information supports maritime analytics at a global level, which enables shipping companies to boost operational efficiency, competitiveness, and sustainability.

Big data in shipping largely comes from three main sources

AIS signals: communication counts
Introduced in 2003 for enhancing maritime safety, the Automatic Identification System (AIS) now offers high-frequency, real-time positioning and sailing patterns for nearly the entire global commercial fleet. This development is considered to have been the catalyst for the digitalisation era in the shipping industry.

Data from the AIS is transmitted every six minutes, and on request, which creates a huge amount of data to process and analyse, when dealing with a whole fleet.
Weather grids: data at its most granular
Maritime weather forecasts are critical for the safety and efficiency of global shipping operations. Weather providers use a grid system, which spans the Earth's surface. This expansive grid delivers detailed weather data at specific points across the globe. The distance between these grid points vary with the scale and design of the model, but in shipping they can be around 10 km apart, in both north/south and east/west directions. Weather forecasts are typically provided on the hour, at each grid point, based on ocean currents, wave height, wind speed, air pressure, visibility and more. This level of granular data is the fuel that powers the evolution of AI, enabling smarter operational decisions.
High frequency data: every move matters
High Frequency Data (HFD) provides continuous updates on vessel performance based on data collected via sensors all around the ship. Every nautical detail, including fuel consumption, speed and position, are measured multiple times a minute. This level of data is significant for a single vessel, but it’s when it’s multiplied across a large fleet, that data really gets big.

The power of complete perspective

Big data in shipping has seen a shift in how the industry perceives and uses data. Traditionally, shipping companies would focus on the data produced by one vessel. The big data revolution now sees data aggregated and analysed across all vessels and all fleets, across the globe. This requires a new generation of technology and expertise, that is able to extract value from massive volumes of a wide variety of data, by enabling high-velocity capturing, discovery and analysis.

Challenges of big data and solutions

Harnessing the power of big data poses a number of challenges for shipping companies. Researchers have characterised these challenges as the ‘4Vs’: Volume, Velocity, Variety and Veracity.

Volume

In the era of digitisation, shipping companies are producing unprecedented volumes of data from a diverse range of sources. Conventional storage and processing systems simply can’t cope, which raises a number of challenges…

Storage: traditional databases are not suited for handling big data, as they have limited scalability and can be expensive to maintain.
Managing access: big data has created a complex data landscape, which can make it difficult for all stakeholders to access and manage data from different sources.
Processing: growing data complexity requires efficient processing.
Cost: managing large data volumes can be expensive, as it requires specialised infrastructure and tools.
Data quality: extensive data generation can compromise quality and accuracy. This runs the risk of errors or biases in data analysis, resulting in inaccurate or incomplete insights.
Data governance: Managing large data volumes raises governance concerns, including data privacy, security, compliance, and access control.
Specific skill sets: There is a shortage of data engineers and data scientists, across all sectors, which poses a challenge as maritime companies compete with other industries for the best talent.

The solution

ZeroNorth has cultivated one of the industry’s largest data ecosystems, by gathering vast amounts of data from our customers and third-party sources. When managing this much data, new technologies and highly skilled personnel are needed. We offer a unique combination of data expertise and the latest technology, along with a deep understanding of the marine industry.

Our dedicated infrastructure serves as a central hub, where all data is stored, processed, analysed and then used to power a range of solutions across specialty areas including chartering, voyage, vessel, bunkering and emissions analysis. ZeroNorth’s proficiency, expertise and economies of scale offer substantial cost savings for customers, by reducing the need for infrastructure investments and extensive personnel.

Velocity

The challenges here parallel data volume, but the focus is on the pace of data generation. The surge in real-time systems and the IoT produces data at unprecedented rates, making it difficult to process and analyse in real-time.

Key issues include:

Outdated systems: traditional IT systems don’t have the capability or capacity to swiftly handle these large datasets.
Data ingestion: efficient mechanisms are needed to handle the high-velocity generation of data from various sources.
Data integration: integrating diverse data from multiple sources with varying formats and structures requires standardisation.
Data quality: The rapid data flow poses a risk of errors and inconsistencies, impacting the accuracy of insights derived.
Data storage: High-speed storage systems are needed to cope with fast-paced data generation, surpassing the capabilities of traditional storage.
Security: robust security mechanisms are crucial to protect sensitive data, encompassing secure transfer, access control, and encryption for data privacy and security.
Skills shortage: as with data volume, meeting these challenges requires specialised skills, which are in high demand and short supply.

The solution

Many shipping companies invest considerable time and money in product development and talent acquisition in an effort to process and analyse big data. But they don’t have to. At ZeroNorth, we have proven success in managing the velocity of big data. Our expertise, tools and scalability offer a seamless and cost-effective solution that will elevate your operational efficiency, environmental responsibility and profitability.

One of the many data tools that ZeroNorth offers is our HFD solution. Edge uses the latest technology, ensuring that data from every vessel is not just collected but also normalised, validated and prepared for actionable insights. By establishing a robust infrastructure onboard, we enable quick and cost-effective installation, feeding our advanced analytics.

Variety

Data from vessels is collected from dozens of different devices and in different formats. This diversity of data makes it difficult to store, process and analyse using traditional data management approaches. For individual companies it's not strategically sound to invest time and resources into converting data into a structured format, as there is poor ROI. Moreover, many companies lack the capacity for such tasks, having not accounted for this kind of extensive work in their business plans.

Key challenges are:

Standardisation: one of the greatest challenges of big data is normalising the volumes of unstructured and disorganised data
Data silos: different departments and stakeholders in shipping often create and store data using different formats, resulting in data silos. This can make it difficult to collate relevant data for analysis and decision-making.
Data quality: unstructured data carries the risk of errors and biases. Many companies lack processes for quality assurance. This makes it challenging to correct and detect data issues, which can impact the reliability of analysis results and overall data quality.
Skill shortage: specialised skills are essential for effective analysis, but shipping companies often struggle to find employees with the right expertise.
Security and privacy: integrating and analysing a range of data sources needs to be done with caution to protect sensitive information and comply with privacy regulations, especially when dealing with unstructured data. This in itself poses challenges as it involves a lot of work.

The solution

Every data challenge presents an opportunity - you just need the right tools, technologies and talent. At ZeroNorth we've crafted an ecosystem where these three things perfectly align to provide automated data validation, which ensures standardised, high quality data.

Our technology is not a one-off solution. It’s a scalable, cost-effective, hassle-free asset for all our customers. In addition, our integrated data governance enables us to ingest and seamlessly blend diverse data types – including data that comes from other software companies. For example, our Edge solution can integrate with third-party software and HFD monitoring systems, and then aggregate the data, presenting it in a concise and organised manner within a single database. This transparency enables seamless data-sharing and prevents data silos. And you don’t need to worry about privacy regulations. Our robust security measures are certified and battle-tested, to ensure they follow the highest standards.

Veracity

Veracity refers to the quality, accuracy, integrity and credibility of data. Simply put, can you trust it? If data is incomplete it can cause more confusion than insights. As most shipping companies don't have the processes in place to deal with veracity, this can lead to incorrect analysis and unsound decisions about the vessel and voyage.

Key challenges include:

Data quality: ensuring accurate, consistent, and complete data involves understanding data sources, collection methods and management processes.
Data integration: merging data from different sources can be complex due to varying formats, structures, and standards. This can lead to data silos, which in turn leads to inconsistencies and errors.
Data cleaning: cleaning data improves data quality, but can be time-consuming.
Data privacy: safeguarding sensitive data and ensuring compliance with regulations is a tough task, which requires the implementation of proper security measures.
Data bias: bias in data occurs when certain elements of a dataset are overweighted or overrepresented, which can lead to poor decision making. Ensuring that data analysis is unbiased requires a clear understanding of the data sources and the analysis techniques used.
Data governance: defining data standards, implementing management processes, and assigning roles and responsibilities for data governance is crucial for ensuring data quality, accuracy and security

The solution

At ZeroNorth, we acknowledge that data is not always perfect and we continuously invest in our solutions to overcome this challenge. If uncertainty around data is accounted for, it is still possible to make smart decisions. Ultimately, it’s about making the best decisions using the best available data, both of which require a specific skill set. This is something we heavily invest in at ZeroNorth. One of our largest R&D investments to-date was our new Fuel Model, which involved a team of more than 25 data scientists, data engineers, software engineers and naval architects. It now uses more than 1.2 billion data points, which generates 34% more accurate fuel consumption predictions when compared to existing solutions and the current industry standard.

V for value

With the right skills and solutions, these four challenges represent real opportunities to create the fifth V: value.

Whether you're a maritime giant or a small fleet owner, the potential value of big data is huge as it offers shipping companies full transparency over vessel performance and marine fuel procurement and empowers them to make smarter decisions and identify new opportunities.

Let ZeroNorth's expertise and centralised services guide you through the complexities and uncertainties of maritime data, and unlock the true value of big data – get in touch to find out more.

Whether you're a maritime giant or a small fleet owner, big data provides a holistic view that transcends individual vessel analytics. It's all about effectively managing and processing data, and transforming it into something of real value.