Streaming with Apache Kafka FAQ

General

What is OCI Streaming with Apache Kafka?

OCI Streaming with Apache Kafka is a managed service that enables you to build real-time data streaming applications using Apache Kafka on Oracle Cloud Infrastructure (OCI). It provides a fully managed Kafka environment, eliminating the need to manage underlying infrastructure.

What are the primary use cases for OCI Streaming with Apache Kafka?

Typical use cases include real-time AI training data foundation, AI agent and multi-agent system orchestration, real-time data analysis, change data capture (CDC), user behavior analytics, fraud detection, metrics and log ingestion, and many more. It can handle high-throughput, low-latency data ingestion and processing use cases.

What are the available cluster types in OCI Streaming with Apache Kafka?

There are two cluster types: starter and high availability (HA). A starter cluster can have between 1 and 30 brokers. HA clusters are recommended for all production workloads with high availability. HA clusters require a minimum of 3 broker nodes across availability domains (ADs) or fault domains (FDs) with a maximum of 30 brokers.

What versions of Apache Kafka are supported?

Our managed Kafka service supports Apache Kafka version 3.7.0, 3.6.1, and 3.6.0. You can find the full list of supported versions in the documentation.

Does OCI Streaming with Apache Kafka support Kafka Raft (KRaft) mode?

Currently, OCI Streaming with Apache Kafka supports only ZooKeeper for managing Kafka metadata and cluster coordination. KRaft mode, which enables Kafka to manage metadata natively without ZooKeeper, is not yet supported. However, KRaft support is planned for a future release, enabling a simplified and more resilient Kafka architecture.

Are Apache Kafka APIs compatible with OCI Streaming with Apache Kafka?

Yes, all data plane and admin APIs are natively supported by OCI Streaming with Apache Kafka.

Can I choose different processor types for OCI Streaming with Apache Kafka?

Currently, the service supports ARM A1 shapes by default. If you need other processor families, such as AMD or Intel, please contact OCI Support to discuss additional options and configurations.

What is the default broker configuration for a cluster?

Unless otherwise specified, OCI Streaming with Apache Kafka uses the defaults specified by the open source version of Apache Kafka. Check the documented default settings for both cluster types.

Can I change the default broker configurations or upload a cluster configuration to OCI Streaming with Apache Kafka?

Yes, OCI Streaming with Apache Kafka allows you to create and apply custom configurations to both new and existing clusters. Currently, custom configurations can be created and versioned using the CLI, which can then be applied when creating a cluster. In the OCI Console, you can select either the default or a custom configuration during cluster setup. For more information on custom configurations, see the configuration documentation.

What configuration properties am I able to customize?

See the documentation on configuration properties you can customize.

Security

Is my data encrypted and protected?

Security is fundamental to OCI Streaming with Apache Kafka. Data movement is protected with TLS 1.2 and data at rest is protected with Block Volume encryption. Network access controls, user authentication (SASL_SCRAM, mTLS), and authorization (Kafka ACL) settings offer further layers of protection to safeguard your data and service access.

Can I restrict access to specific Kafka topics?

Yes, you can use Kafka Access Control Lists (ACLs) to restrict access to specific topics, consumer groups, and administrative operations. This helps ensure that only authorized clients can access Kafka resources.

What authentication methods are available for OCI Streaming with Apache Kafka?

OCI Streaming with Apache Kafka supports multiple authentication methods to secure access to your Kafka cluster. Users can authenticate via Simple Authentication and Security Layer/Salted Challenge Response Authentication Mechanism (SASL/SCRAM) or Mutual TLS (mTLS) for secure communication and access control. These options provide robust security measures to help ensure that only authenticated users can interact with the Kafka clusters.

Does OCI Streaming with Apache Kafka support Lightweight Direct Access Protocol (LDAP) or Kerberos authentication?

At present, the service supports SASL/SCRAM and mTLS for authentication. In the future, OCI Streaming with Apache Kafka will support OCI Identity and Access Management (IAM), which will enable integration with Active Directory and LDAP through federated identity providers and identity synchronization for streamlined authentication and user management.

Does OCI Streaming support virtual cloud network (VCN) peering for secure, private access?

Yes, you can configure your Kafka clusters within private subnets and use VCN peering to securely connect from other OCI VCNs or your on-premises network, depending on your architecture requirements.

Performance and scalability

What are the throughput and scalability limits of OCI Streaming with Apache Kafka?

OCI Streaming with Apache Kafka supports scalable data ingestion and processing, with customizable broker configurations to meet high-throughput requirements. You can add up to 30 brokers in a cluster. For further details, refer to the documentation.

How does OCI Streaming handle load balancing and fault tolerance?

For high availability clusters, OCI Streaming with Kafka automatically distributes brokers across multiple ADs or FDs for resilience. The service also provides automated load balancing and partitioning across brokers.

Can I dynamically scale my Kafka cluster?

Yes, you can scale your Kafka cluster within the broker count limits (1–30 brokers). This allows you to adjust resources to match workload demands without disrupting service. For greater broker requirements, contact support.

Connectivity

Can I access OCI Streaming with Apache Kafka from outside the Oracle Cloud?

Currently, public connectivity to OCI Streaming with Apache Kafka is not available. Access to Kafka clusters is restricted to OCI VCNs. To connect external producers and consumers, you can set up private connectivity solutions, such as OCI FastConnect or VPN, to securely extend your on-premises network or other cloud environments to OCI.

What configurations are needed to connect Kafka clients from my on-premises network?

For on-premises connectivity, you can set up an OCI FastConnect or VPN connection to securely access your Kafka cluster hosted on OCI.

Does OCI Streaming with Apache Kafka support replication between clusters (Active/Active and Active/Standby)?

Yes, OCI Streaming with Apache Kafka supports intercluster replication, which can be configured for Active/Active or Active/Standby setups using MirrorMaker 2. This allows for flexible data replication strategies across clusters, enabling disaster recovery and data synchronization. By default, data within an HA configuration is replicated within the same region for resiliency. You can use MirrorMaker 2 to customize additional cross-region or cross-cluster replication based on your specific requirements.

Can I migrate data within my existing Apache Kafka cluster to OCI Streaming with Apache Kafka?

Yes, you can use tools such as MirrorMaker 2 to replicate data from clusters into OCI Streaming with Apache Kafka.

Does OCI Streaming with Apache Kafka support change data capture (CDC) with Debezium?

Yes, OCI Streaming with Apache Kafka supports CDC using Debezium, along with any other Kafka connectors. Currently, users are responsible for managing and maintaining their own connectors for CDC. However, a fully managed Kafka Connect service, which will simplify connector management, is planned for a future release.

Operations and monitoring

How do I monitor my Kafka cluster on OCI?

You can monitor cluster- and broker-level metrics from the OCI Monitoring console with namespace oci_kafka. For details, refer to the documentation. Additional, high-cardinality metrics (topic- and partition-level) will be available soon.

Is logging available for OCI Streaming with Apache Kafka?

Currently, OCI Streaming with Apache Kafka offers only audit logs. Broker or service logs will be offered soon.

How do I manage topic configurations and replication?

Topic configurations, partitions, and replication settings can be managed using the Kafka CLI, SDKs, or Kafka APIs. Currently, the OCI Console does not support direct management of these configurations. You can set the replication factor through Kafka APIs to meet your fault tolerance and availability needs.

Is there a UI for administering the Kafka cluster in OCI?

Currently, OCI Streaming with Apache Kafka does not provide a native UI for cluster administration. However, you can use your own third-party tools—such as Kafbat, AKHQ, and similar—to manage and monitor your Kafka clusters. These tools can be deployed within OCI, allowing you to view and administer the cluster seamlessly within your own environment.

Pricing and billing

How is OCI Streaming with Apache Kafka priced?

The pricing for OCI Streaming with Apache Kafka is based on the underlying infrastructure. You can configure the cluster by selecting the amount of Kafka brokers, cores, memory, and storage. The associated infrastructure costs are pass-through charges based on the OCI pricing list (see A1 flex shapes for OCPU pricing). Additionally, a service fee of $0.10 per OCPU per hour applies based on the number of OCPUs provisioned. For clusters using the Arm A1 shape, the service fee is discounted by 50%, resulting in a $0.05 charge per OCPU per hour.

Are there additional costs for data storage in OCI Streaming?

Yes, data storage costs are based on the Block Volume storage that you select during cluster provisioning. You can choose between 50 GB and 5 TB of storage per broker. The associated storage cost is a pass-through expense based on OCI Block Volume pricing. For instance, if you need 15 TB of storage, you will provision 3 brokers with 5 TB each, totaling 15 TB of storage for the cluster.

Integration and compatibility

Does OCI Streaming with Apache Kafka support Kafka Connect?

Currently, OCI Streaming with Apache Kafka does not include a managed Kafka Connect service, but you can deploy Kafka Connect on OCI Compute instances and integrate it with your Kafka clusters.

Can I use OCI Streaming with Apache Kafka with Oracle GoldenGate for change data capture?

Yes, you can integrate Oracle GoldenGate with OCI Streaming with Apache Kafka to capture and stream changes from databases to Kafka, supporting various data replication and analytics use cases.

How does OCI Streaming with Apache Kafka integrate with OCI Data Flow?

OCI Data Flow can be used to process data from Kafka topics directly, allowing you to build and execute Spark applications for data transformation, analytics, and storage.

Is a Schema Registry available with OCI Streaming with Apache Kafka?

Yes, the service currently supports integration with open source Schema Registry options, allowing you to manage and store schemas for your data streams. A fully managed Schema Registry offering is in development and expected in a future release, simplifying schema management within the OCI environment.