New capabilities enable customers to ingest, manage, and share data in near real-time while leveraging the scale and cost efficiency of data lakes.
starburst, the data lake analytics platform, announced new features at AWS re:Invent 2023 that enable organizations to build and scale innovative data applications without sacrificing performance or cost. As interest in building artificial intelligence (AI)-driven data applications increases, customers need to establish a solid data platform. New features in Starburst Galaxy help customers simplify development in data lakes by consolidating data ingestion, data governance, and data sharing into a single platform.
Recommended AI news: Riding the Generative AI Hype, CDP Will Need New Definition in 2024
Interactive applications often require the scalability and cost efficiency of a data lake, but building and maintaining that data lake can be complex and time-consuming for data teams. To overcome these challenges, Starburst has added support for:
- Near real-time analysis with streaming ingestion: With streaming ingest, customers can leverage Kafka to hydrate their data lake in near real-time, ensuring their applications have the most up-to-date insights for their users. In the future, support for fully managed solutions such as Confluent Cloud is also planned.
- Automated data governance: As new data arrives at the lake, machine learning models in Gravity, Starburst Galaxy’s universal discovery, governance, and sharing layer, automatically apply classification to specific categories. Depending on the class, Gravity applies policies that allow or restrict access. This automation is especially useful for teams that work with sensitive data such as personally identifiable information (PII). As soon as PII lands in the lake, Gravity is smart enough to identify and restrict access to that data.
- Automatic data maintenance: New automations allow customers to easily optimize their data lakes by abstracting common management tasks such as data compression and data vacuuming. As the amount and complexity of data in a data lake grows, users can now maintain warehouse-like performance without adding brittle manual processes.
- Universal data sharing with built-in observability: Gravity allows users to easily package data sets, regardless of source, format, or cloud provider, into shareable data products to power end-user applications. New features enable users to securely share these high-quality data products with third parties such as partners, suppliers, and customers.
- AI-powered self-service analytics: Not only are data lakes notoriously difficult to manage, data teams are largely understaffed. New AI-powered experiences in Galaxy, such as text-to-SQL processing, enable data teams to offload basic exploratory analysis to business users, freeing up time to build and scale data pipelines. will be done.
Recommended AI news: Clario’s new AI-powered ECG quality score tool enhances cardiac safety assessment
“Data-intensive initiatives like AI require a solid data foundation to be successful,” he said. Justin Borgman, co-founder and CEO of Starburst. “We provide that foundation and enable our customers to quickly access and analyze all their data to scale their applications from the first 100 users to the first 1,000 users and beyond. High concurrency ensures optimal performance even as data volumes grow exponentially. Starburst’s new streaming ingest, data maintenance and governance automation, and data sharing capabilities enable teams to leverage data lakes. It’s super easy to build, deploy, and scale applications on top of it.”
Halliburton is already leveraging this foundation. “After building a high-quality data product at Starburst, he saw an opportunity to use LLM to help with that process,” he said. Fahad Ahmad, Data Science Leader at Halliburton. “Previously, it would take two to three weeks to get an answer to an ad-hoc question. By incorporating LLM into Starburst’s data product architecture, data consumers can ask questions in plain language and translate them into SQL , and get answers instantly.”
Recommended AI news: Cyara strengthens AI-based chatbot optimization capabilities with acquisition of QBox
Starburst’s status as an Amazon Web Services (AWS) Data and Analytics Competency Partner means you can rest assured that these capabilities are available on the fastest hardware AWS has to offer, including AWS Graviton3 and the newly launched Amazon Simple Storage Service. To do. (Amazon S3) is a zonal storage class that integrates seamlessly with core tools like AWS QuickSight and new tools like Amazon Bedrock.
[To share your insights with us, please write to [email protected]]