Python Crawling: Selenium driver error # --user-data-dir # --headless # selenium.common.exceptions.SessionNotCreatedException: Message: session not created: probably user data directory is already in use, please specify a unique value for --user-data-dir

Python의 Selenium 라이브러리를 통해 크롤링을 진행하던 중 아래의 에러를 만났다.

 

selenium.common.exceptions.SessionNotCreatedException: Message: session not created: probably user data directory is already in use, please specify a unique value for --user-data-dir argument, or don't use --user-data-dir

 

로컬에서는 이상없이 동작하던 스크립트가 remote 서버에서는 위와 같은 에러를 마주했다.

위 에러를 검색하면 stackoverflow나 AI(ChatGPT, Claude 등)는 다음과 같은 해결책을 제시해준다.

import tempfile # <- solution
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from settings import driver_path


options = webdriver.ChromeOptions()
temp_profile_dir = tempfile.mkdtemp() # <- solution
options.add_argument(f"--user-data-dir={temp_profile_dir}") # <- solution
service = webdriver.ChromeService(driver_path)
driver = webdriver.Chrome(options, service)

 

위와 같이 --user-data-dir 으로 사용할 폴더를 직접 만들어서 options에 추가하라는 것이다.

breakpoint()를 잡고 확인해보면 위 과정을 거치면서 임시파일도 제대로 생성되는데 같은 오류가 여전히 발생한다.

 

문제는 ssh로 접속한 remote 서버에는 GUI를 실행시킬 환경이 되지 못해서 

options.add_argument('--headless')

 

위 조건을 추가해야 한다는 것이다.

 

해결은 위 옵션을 추가함으로써 되는데

왜 Selenium은 --user-data-dir 와 관련된 에러를 띄웠을까?

 

AI에 물은 결과는 다음과 같다. (정확하지 않을 수 있으니 참고만)

The error message about the user data directory being in use is a bit misleading in your situation. In your SSH environment without a display, if you don’t add the --headless flag, Chrome attempts to launch in full (GUI) mode. Since there’s no display (or X server) available, Chrome fails to start properly. During that failed initialization, it may still create or lock some files in the user data directory, leading Chrome to report that the directory is "already in use."

When you add the --headless flag, Chrome understands that it should run without any GUI and avoids trying to create a display or lock files in a way that conflicts with itself. This allows Chrome to initialize properly, so the error no longer appears.

In summary, while the error message points to a user data directory conflict, the underlying cause was that Chrome was trying (and failing) to launch in a non-headless (GUI) mode in an environment without a monitor. The --headless flag tells Chrome to bypass the GUI initialization, which resolves the issue.