AWS EC2에 카프카 클러스터 구성시 에러

요약 : 돈을 아끼기위해 프리티어스펙의 EC2로 인스턴스를 생성했을경우 메모리가 1G 할당된다.
문제는 카프카 브로커 설치 이후 주키퍼를 실행할때 512MB를 먹고 시작하며.. 이후 카프카가 1G를 먹고 실행하려던 중 에러가 발생한다.
금액을 지불하고 인스턴스 스펙을 올리던가 JVM메모리를 조정하여야 한다.

에러 로그

$ kafka-topics --zookeeper ...생략... --create --topic first-test --partitions 3 --replication-factor 3

OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Error while executing topic command : Replication factor: 3 larger than available brokers: 0.
[2020-05-13 09:24:03,192] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 0.
 (kafka.admin.TopicCommand$)

상태확인

• Active: failed

  $ sudo systemctl status confluent-kafka
  

● confluent-kafka.service - Apache Kafka - broker
Loaded: loaded (/usr/lib/systemd/system/confluent-kafka.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2020-05-13 09:26:45 UTC; 5s ago
Docs: http://docs.confluent.io/
Process: 1772 ExecStart=/usr/bin/kafka-server-start /etc/kafka/server.properties (code=exited, status=1/FAILURE)
Main PID: 1772 (code=exited, status=1/FAILURE)

May 13 09:26:45 ip-172-31-42-242.ap-northeast-2.compute.internal kafka-server-start[1772]: OpenJDK 64-Bit Server VM warning: If the number of processors is ex...ads=N
May 13 09:26:45 ip-172-31-42-242.ap-northeast-2.compute.internal kafka-server-start[1772]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000...o=12)
May 13 09:26:45 ip-172-31-42-242.ap-northeast-2.compute.internal kafka-server-start[1772]: #
May 13 09:26:45 ip-172-31-42-242.ap-northeast-2.compute.internal kafka-server-start[1772]: # There is insufficient memory for the Java Runtime Environment to ...inue.
May 13 09:26:45 ip-172-31-42-242.ap-northeast-2.compute.internal kafka-server-start[1772]: # Native memory allocation (mmap) failed to map 1073741824 bytes fo...mory.
May 13 09:26:45 ip-172-31-42-242.ap-northeast-2.compute.internal kafka-server-start[1772]: # An error report file with more information is saved as:
May 13 09:26:45 ip-172-31-42-242.ap-northeast-2.compute.internal kafka-server-start[1772]: # /tmp/hs_err_pid1772.log
May 13 09:26:45 ip-172-31-42-242.ap-northeast-2.compute.internal systemd[1]: confluent-kafka.service: main process exited, code=exited, status=1/FAILURE
May 13 09:26:45 ip-172-31-42-242.ap-northeast-2.compute.internal systemd[1]: Unit confluent-kafka.service entered failed state.
May 13 09:26:45 ip-172-31-42-242.ap-northeast-2.compute.internal systemd[1]: confluent-kafka.service failed.

/tmp/hs_err_pid1772.log 파일을 확인해보니 메모리 할당 오류인 것을 확인.

### 메모리 수정

카프카 메모리 수정.

$ sudo vi /usr/bin/kafka-server-start

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
# export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M"
fi

---

주키퍼 메모리 수정

$ sudo vi /usr/bin/zookeeper-server-start

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
# export KAFKA_HEAP_OPTS="-Xmx512M -Xms512M"
export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M"
fi

### 재시작

재시작

sudo systemctl restart confluent-zookeeper
sudo systemctl restart confluent-kafka

상태확인

sudo systemctl status confluent-zookeeper
sudo systemctl status confluent-kafka
```