|
1 | 1 | package com.linkedin.davinci.consumer; |
2 | 2 |
|
3 | | -import com.linkedin.venice.annotation.Experimental; |
4 | 3 | import com.linkedin.venice.pubsub.api.PubSubMessage; |
5 | 4 | import java.util.Collection; |
6 | 5 | import java.util.Set; |
7 | 6 | import java.util.concurrent.CompletableFuture; |
8 | 7 |
|
9 | 8 |
|
10 | 9 | /** |
11 | | - * This interface is meant for users where local state must be built off of the entirety of a venice data set |
12 | | - * (i.e. Non-idempotent event ingestion), rather than dealing with an event at a time. THIS IS EXPENSIVE. |
13 | | - * It's highly recommended that users use the {@link VeniceChangelogConsumer} interface as a means to consume Venice |
14 | | - * Change capture data. |
| 10 | + * This interface is meant for users where local state must be built off of the entirety of a Venice data set. |
15 | 11 | * |
16 | | - * Implementations of this interface rely on access to a compacted view to the data and scanning the entirety of that |
17 | | - * compacted view initial calls to poll(). This is the only supported pattern with this interface today. {@link VeniceChangelogConsumer} |
18 | | - * enables finer control. This interface is intentionally limited as implementors rely on local checkpointing and |
19 | | - * maintenance of state which might be easily corrupted with byzantine seek() calls. |
20 | | - * @param <K> |
21 | | - * @param <V> |
| 12 | + * This interface provides automatic state management with local checkpointing and efficient data access |
| 13 | + * through local compaction. It eliminates the need for manual checkpoint management and improves restart performance. |
| 14 | + * |
| 15 | + * KEY BENEFITS: |
| 16 | + * - Automatic State Management: No need to manually manage checkpoints. |
| 17 | + * The client handles all state management automatically. |
| 18 | + * - Efficient Restart: Resumes from the last checkpoint on restart, consuming only new changes since the last |
| 19 | + * Kafka checkpoint. This reduces recovery time and eliminates the need to re-consume every event from Kafka |
| 20 | + * on restart. |
| 21 | + * - Local Compaction: All data is compacted locally, providing efficient access to the current state without consuming |
| 22 | + * duplicate events. |
| 23 | + * - Fast Bootstrap on Fresh Nodes: On fresh nodes without local state, obtains a complete data snapshot from existing |
| 24 | + * nodes instead of consuming evert Kafka event (requires blob transfer to be enabled). |
| 25 | + * |
| 26 | + * This interface intentionally does not expose seek() operations for simplicity. |
| 27 | + * For more fine-grained control over seeking, see {@link VeniceChangelogConsumer}. |
| 28 | + * |
| 29 | + * @param <K> Key type |
| 30 | + * @param <V> Value type |
22 | 31 | */ |
23 | | -@Experimental |
24 | 32 | public interface BootstrappingVeniceChangelogConsumer<K, V> { |
25 | 33 | /** |
26 | | - * Start performs both a topic subscription and catch up. The client will look at the latest offset in the server and |
27 | | - * sync bootstrap data up to that point in changes. Once that is done for all partitions, the future will complete. |
28 | | - * |
29 | | - * NOTE: This future may take some time to complete depending on how much data needs to be ingested in order to catch |
30 | | - * up with the time that this client started. |
31 | | - * |
32 | | - * NOTE: In the experimental client, the future will complete when there is at least one message to be polled. |
33 | | - * We don't wait for all partitions to catch up, as loading every message into a buffer will result in an |
34 | | - * Out Of Memory error. Instead, use the {@link #isCaughtUp()} method to determine once all subscribed partitions have |
35 | | - * caught up. |
| 34 | + * Starts the consumer by subscribing to the specified partitions. On restart, the client automatically resumes |
| 35 | + * from the last checkpoint. On fresh start, it begins from the beginning of the topic or leverages blob transfer |
| 36 | + * if available. |
36 | 37 | * |
37 | | - * NOTE: In the experimental client, if you pass in an empty set, it will subscribe to all partitions for the store |
| 38 | + * The returned future completes when there is at least one message available to be polled. |
| 39 | + * Use {@link #isCaughtUp()} to determine when all subscribed partitions have caught up to the latest offset. |
38 | 40 | * |
39 | | - * @param partitions which partition id's to catch up with |
40 | | - * @return a future that completes once catch up is complete for all passed in partitions. |
| 41 | + * @param partitions Set of partition IDs to subscribe to. Pass empty set to subscribe to all partitions. |
| 42 | + * @return A future that completes when at least one message is available to poll. |
41 | 43 | */ |
42 | 44 | CompletableFuture<Void> start(Set<Integer> partitions); |
43 | 45 |
|
| 46 | + /** |
| 47 | + * Subscribes to every partition for the Venice store. See {@link #start(Set)} for more information. |
| 48 | + */ |
44 | 49 | CompletableFuture<Void> start(); |
45 | 50 |
|
46 | 51 | void stop() throws Exception; |
47 | 52 |
|
48 | 53 | /** |
49 | | - * polls for the next batch of change events. The first records returned following calling 'start()' will be from the bootstrap state. |
50 | | - * Once this state is consumed, subsequent calls to poll will be based off of recent updates to the Venice store. |
| 54 | + * Polls for the next batch of change events. The first records returned after calling {@link #start(Set)} will be from the |
| 55 | + * local compacted state. Once the local state is fully consumed, subsequent calls will return |
| 56 | + * real-time updates made to the Venice store. |
51 | 57 | * |
52 | | - * In the experimental client, records will be returned in batches configured to the MAX_BUFFER_SIZE. So the initial |
53 | | - * calls to poll will be from records from the bootstrap state, until the partitions have caught up. |
54 | | - * Additionally, if the buffer hits the MAX_BUFFER_SIZE before the timeout is hit, poll will return immediately. |
| 58 | + * Records are returned in batches up to the configured MAX_BUFFER_SIZE. This method will return immediately if: |
| 59 | + * 1. The buffer reaches MAX_BUFFER_SIZE before the timeout expires, OR |
| 60 | + * 2. The timeout is reached |
55 | 61 | * |
56 | | - * If the PubSubMessage came from disk (after restart), the following fields will be set to sentinel values since |
57 | | - * record metadata information is not available to reduce disk utilization: |
| 62 | + * NOTE: If the PubSubMessage came from disk (after restart), the following fields will be set to sentinel values |
| 63 | + * since record metadata information is not persisted to reduce disk utilization: |
58 | 64 | * - PubSubMessageTime |
59 | 65 | * - Position |
60 | 66 | * |
61 | | - * @param timeoutInMs Maximum timeout of the poll invocation |
62 | | - * @return a collection of Venice PubSubMessages |
| 67 | + * @param timeoutInMs Maximum timeout of the poll invocation in milliseconds |
| 68 | + * @return A collection of Venice PubSubMessages containing change events |
63 | 69 | */ |
64 | 70 | Collection<PubSubMessage<K, ChangeEvent<V>, VeniceChangeCoordinate>> poll(long timeoutInMs); |
65 | 71 |
|
66 | 72 | /** |
67 | | - * In the experimental client, once this becomes true it will stay true even if we start to lag after the |
68 | | - * bootstrapping phase. |
69 | | - * @return True if all subscribed partitions have caught up. |
| 73 | + * Indicates whether all subscribed partitions have caught up to the latest offset at the time of subscription. |
| 74 | + * Once this becomes true, it will remain true even if the consumer begins to lag later on. |
| 75 | + * |
| 76 | + * This is for determining when the initial bootstrap phase has completed and the consumer has transitioned |
| 77 | + * to consuming real-time events. |
| 78 | + * |
| 79 | + * @return True if all subscribed partitions have caught up to their target offsets. |
70 | 80 | */ |
71 | 81 | boolean isCaughtUp(); |
72 | 82 |
|
|
0 commit comments