Arnaud Van der Poorten on
For any modern production facility, staying competitive requires a focus on operational efficiency and data-driven decision-making. How we store, process, and share data plays a major role in achieving this.
Find out why we chose Parquet as a format for exporting data from Factry Historian, and how it delivers significant benefits in both data storage and processing speed.
Your industrial facility generates data from countless sources: PLCs, SCADA and DCS systems, standalone sensors like energy metres, and various other devices, all contributing to a vast ocean of information. At Factry, our goal is to make this process data easy to access, efficient to handle, and valuable for everyone — whether you’re a process engineer, production supervisor, or IT expert.
With Factry Historian, you can capture process data in real time to make smarter decisions. But capturing production data is just the beginning. Factry Historian also enables people to analyse the data in a frictionless, visual and user-friendly way. Additionally, Factry Historian provides an efficient way to export both the data and its metadata.
This is where Apache Parquet comes in.
Parquet is a columnar storage format that shines when handling industrial process data, such as time-series data, event streams, or sensor readings. Unlike traditional row-based storage formats such as CSV or relational tables, Parquet’s columnar approach is designed to make data compression and query efficiency incredibly effective.
It ensures that your data can be stored with minimal footprint while still being readily available for fast, responsive queries.
Data from factories is often massive and continuous: imagine hundreds of sensors on a production line sending data on temperature, pressure, and speed readings every single second.
When exporting this data from Factry Historian as Parquet files, you can benefit from two main types of efficiency:
Parquet’s columnar structure allows for powerful compression techniques. Instead of storing repetitive data across rows, Parquet compresses each column separately, achieving significant storage space savings. This is especially useful in industries where data collection is happening 24/7, resulting in terabytes of production data every year.
By reducing the storage footprint, Parquet helps lower costs, no matter if the data is stored on-premises or in the cloud.
Parquet allows for faster analytics by reading only the columns needed for a query. Imagine needing to analyse power consumption from hundreds of production lines over several months. With Parquet, you can skip irrelevant data (e.g. temperature or pressure) and quickly focus on the relevant columns.
This leads to faster, more efficient queries that are less costly in terms of computing power. Such efficiency translates to quicker insights, enabling teams to respond promptly to production issues or optimise processes.
Industrial environments are inherently complex, and data often resides in different silos. And at Factry, we aim to help break down these silos. Parquet, as a free, open-source, and widely supported format, is key to this vision.
The beauty of Parquet lies in its ability to seamlessly integrate with a wide array of tools without the need for complex transformations or proprietary connectors. Data exported in Parquet can easily be ingested by many different tools, whether you are using Apache Spark for big data analysis, Tableau for visualisation, Power BI for business intelligence, or simply running machine learning models in Python.
It’s like having a key that fits every door.
For Factry Historian users, this means your data isn’t locked into a single application or vendor — it’s your data, fully portable, and ready to use in the tools that best fits your needs. This interoperability makes collaboration between departments and roles (from operators to data scientists) more fluid and effective, enabling a truly data-driven culture across your factory.
At Factry, we value an open and transparent approach to technology. Just as we designed Factry Historian to be intuitive and accessible, we also believe that the data collected should not be locked in a proprietary format that limits its potential use.
By leveraging free, open-source standards like Parquet for exporting data, we ensure that the data remains accessible to our customers — not just now, but in the future as well. This openness empowers our clients to build custom solutions, integrate with other systems, and get value out of it without facing restrictive barriers.
Using Parquet aligns with our focus on simplicity, flexibility, and openness. It’s about giving you the freedom to use your data however you see fit, to turn raw numbers into meaningful actions and insights, and to build a production environment that’s resilient and adaptable.
In summary, Parquet isn’t just another storage format, but a true enabler of efficiency, interoperability, and freedom in the industrial world. By providing the ability to export process data in Parquet format from Factry Historian, we’re ensuring that your data is readily available, in an efficient, industry standard format, to drive smarter decisions from the factory floor to the boardroom.
Factry is committed to keeping your data open, efficient, and accessible. This enables you to focus on what truly matters: optimising production operations, improving quality, and driving innovation. The journey towards data-driven production starts with the right tools, and at Factry, we’re proud to provide them, helping you transform data into tangible process improvements.
Have any questions after reading this article? Get in touch with us.
Steam management has always been an essential aspect of industries such as food and beverage, paper production, chemical processing, and energy generation. With Factry Historian v7.1, steam functions are now fully integrated, making steam calculations straightforward, reliable, and accessible. Here’s how it works.
Is open source software (OSS) the right decision for your production company? In contrast to proprietary software, the possibilities in terms of flexibility and scalability are endless. Yet, not all open source applications are alike.
Factry Historian v7.1 is here, bringing significant performance updates that will make your data processing smoother, faster, and more efficient. This update focuses on speeding up calculations, simplifying asset management, and adding new functionality like steam functions, ensuring your Historian experience is more powerful than ever.
