File: //usr/libexec/kcare/python/kcarectl/doctor.pyo
�
�;jc . @ s d Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l m Z m Z m Z m
Z
m Z m Z m
Z
m Z m Z d d l m Z d d l m Z m Z m Z e r� d d l m Z m Z m Z m Z m Z n d Z d Z d Z d
Z d Z! d Z" d
Z# d Z$ d Z% d� Z& d� Z' d� Z( d e j) f d � � YZ* d e+ f d � � YZ, d � Z- d � Z. d � Z/ d � Z0 d! e+ f d" � � YZ1 d# � Z2 d$ � Z3 e d% � Z4 d& � Z5 e j6 d' � � Z7 d( � Z8 d) � Z9 d* � Z: d+ � Z; d, � Z< d- � Z= d. � Z> d/ � Z? d0 � Z@ d1 � ZA d2 � ZB d3 � ZC d4 � ZD d5 � ZE e1 d6 e5 � e1 d7 e4 d8 g d9 eF �� e1 d: e7 � e1 d; e8 � e1 d< e9 � e1 d= e4 d= g � � e1 d> e: � e1 d? e2 d@ � � e1 dA e2 dB � � e1 dC e2 dD � � e1 dE e2 dF � � e1 dG e2 dH � � e1 dI e2 dJ � � e1 dK e2 dL � � e1 dM e2 dN � � e1 dO e2 dP � � e1 dQ e3 dR dS � � e1 dT e; � e1 dU e3 dV dW � � e1 dX e< � e1 dY e= � g ZG e- dZ d[ jH e$ � � e- d\ d\ � e- d] d^ jH e$ � � e- d_ d` � e. da db � e- dc dd � e/ de df � e/ dg dh � e/ di dj � e/ dk dl � e/ dm dn � e/ do dp � e/ dq dr � e/ ds dt � e/ du dv � e. dw dx � e/ dy dz � e. d{ e � e/ d| d} � e- d~ d � e- d� d� � e- d� d� � e- d� d� � e0 d� e> � e- d� d� � e. d� d� � e0 d� e@ � e- d� d� � e- d� d� � e0 d� eA � e. d� d� � e- d� d� � e0 d� eB � e- d� d� jH e$ � � e. d� e � e. d� d� � e0 d� eC � e0 d� eD � e- d� d� � e- d� d� � e- d� d� � e0 d� eE � e- d� d� � e- d� d� � g, ZI e jJ d� e jK � d� � � ZL e d� � ZM d S(� sO Doctor v2 vitals collector (KPT-5984).
Python port of the kcdoctor.sh (1.0-8) vitals collection: gathers the
same data set into a DataPackage-based archive processed by the patch
server generic uploads pipeline.
Scalar and short values are consolidated into a single ``doctor.json``
(the eportal doctor precedent), one field each, instead of a tiny
archive entry per value; bulk data (logs, command dumps, package lists,
sysfs tunables, crash artifacts) stays as separate archive entries. The
1:1 inventory of collected items lives in
docs/features/KPT-5984-doctor-v2/parity-inventory.md.
i����N( t autht configt delivery_kitt
http_utilst kcaret log_utilst platform_utilst serveridt utils( t run_command( t HTTPErrort URLErrort httplib( t Anyt Callablet Dictt Listt Optionals 2.0-1s /boot/grub2/grub.cfgs /etc/kdump.confs /var/log/dpkg.logs /etc/apt/sources.list*s /etc/yum.repos.d/*s /var/log/libcare/*.logi' s /syss enable*s nr_*s max_*s *cnt*s /sys/kernel/debug/tracing/eventss /sys/kernel/tracing/eventss /sys/devicess
[0-9]*.logs kmsg*logt KcarectlDoctorPackagec B s# e Z d Z d Z e d � � Z RS( s kcarectl-doctors /upload/kcarectl-doctor/c C s t j S( N( R t DOCTOR_REPORT_MAX_SIZE_BYTES( t self( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt max_size; s ( t __name__t
__module__t data_typet
upload_urit propertyR ( ( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyR 7 s t CollectionSpecc B s e Z d Z d � Z d � Z RS( s� A single bulk vitals item ported from kcdoctor.sh.
Collection failures are recorded in errors.log and never abort the
report: an unhandled exception inside the package context manager
would remove the whole archive.
c C s | | _ | | _ d S( N( t arcnamet _collect( R R t collect( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt __init__L s c C sM y | j | | j � Wn/ t k
rH } | j d j | j | � � n Xd S( Ns failed to collect {0}: {1}( R R t Exceptiont log_errort format( R t data_packaget e( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt applyQ s ( R R t __doc__R R% ( ( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyR D s c s � f d � } t | | � S( s4 `run "<cmd>"` kcdoctor.sh helper (no shell features)c s | j | � � d S( N( t
add_stdout( R# R ( t cmd( s, /usr/libexec/kcare/python/kcarectl/doctor.pyR ] s ( R ( R R( R ( ( R( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt run_specY s c s � f d � } t | | � S( s4 `dump "<path>"` kcdoctor.sh helper for regular filesc s | j | d � �d S( Nt src_path( t add_file( R# R ( t path( s, /usr/libexec/kcare/python/kcarectl/doctor.pyR h s ( R ( R R, R ( ( R, s, /usr/libexec/kcare/python/kcarectl/doctor.pyt dump_specd s c s � f d � } t | | � S( s� `dump "<path>"` for /proc and /sys files.
tar reports them as 0-size so they are first read into memory
(see the same approach in anomaly.prepare_kernel_anomaly_report).
c sa t j j � � s, | j d j � � � d St � d � � } | j | d | j � �Wd QXd S( Ns file not found: {0}t rbt
data_bytes( t osR, t existsR! R" t openR+ t read( R# R t f( R, ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyR w s
( R ( R R, R ( ( R, s, /usr/libexec/kcare/python/kcarectl/doctor.pyt proc_dump_speco s c C s
t | | � S( N( R ( R t fn( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt
callable_spec� s t FieldSpecc B s e Z d Z d � Z d � Z RS( s� A single doctor.json field ported from kcdoctor.sh.
Collection failures are recorded in errors.log and never abort the
report; a collector returning None omits the field (e.g. a
distro-specific source file absent on this system).
c C s | | _ | | _ d S( N( t keyR ( R R9 R ( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyR � s c C sc y | j � } Wn0 t k
rB } | j d j | j | � � d SX| d k r_ | | | j <n d S( Ns failed to collect {0}: {1}( R R R! R" R9 t None( R R# t reportt valueR$ ( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyR% � s ( R R R& R R% ( ( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyR8 � s c s � f d � } | S( s� `dump "<path>"` as a doctor.json string field.
Read in binary (non-utf8 safe) and decoded leniently; an absent
distro-specific source file simply omits the field (None).
c sN t j j � � s d St � d � �# } | j � j d d � j � SWd QXd S( NR. s utf-8t replace( R0 R, R1 R: R2 R3 t decodet strip( R4 ( R, ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyR � s ( ( R, R ( ( R, s, /usr/libexec/kcare/python/kcarectl/doctor.pyt
read_field� s c s. t j t j | � � � � � f d � } | S( s� `grep <pattern> <path>` as a doctor.json list-of-lines field.
Match on bytes (like collect_apt_sources) so a non-utf8 byte cannot
lose a line; the matched lines are decoded leniently for the JSON.
c sr t j j � � s d St � d � � } | j � j � } Wd QXg | D]' } � j | � rG | j d d � ^ qG S( NR. s utf-8R= ( R0 R, R1 R: R2 R3 t
splitlinest searchR> ( R4 t linest line( R, t regex( s, /usr/libexec/kcare/python/kcarectl/doctor.pyR � s
( t ret compileR t bstr( R, t patternR ( ( R, RE s, /usr/libexec/kcare/python/kcarectl/doctor.pyt
grep_field� s
c s � � f d � } | S( s@ `run "<cmd>"` captured as a doctor.json field instead of a file.c sG t � d t d t �\ } } } | p* d j � } � rC | j � S| S( Nt catch_stdoutt catch_stderrt ( R t TrueR? RA ( t _t stdoutt text( t argvt as_lines( s, /usr/libexec/kcare/python/kcarectl/doctor.pyR � s !( ( RR RS R ( ( RR RS s, /usr/libexec/kcare/python/kcarectl/doctor.pyt cmd_field� s c C s t S( N( t DOCTOR_VERSION( ( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt collect_doctor_version� s c C sN y5 t j t j d � � } t j | j � � j � SWn t k
rI d SXd S( s� the public IP of the machine as seen by the patch server.
Cached: the main_ip and server_id fields share one HTTP roundtrip
per run.
t myipt NAN( R t urlopenR t get_patch_server_urlt nstrR3 R? R ( t response( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt get_main_ip� s
c C s7 t j � } | s* t � j d d � } n t j | � S( Nt .RO ( R t get_serveridR] R= R R[ ( t server_id( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt collect_server_id� s c C s
t j � S( N( R t get_kernel_hash( ( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt collect_kernel_id� s c C s\ i } xO d D]G } t d d | g d t d t �\ } } } | pG d
j � | | <q
W| S( Nt at rt mt pt ot unamet -RK RL RM ( Rd Re Rf Rg Rh ( R RN R? ( t resultt flagRO RP ( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt
collect_uname� s
+c C s~ t j j t � s d St t d � � } | j � j d d � } Wd QXg | j � D]' } d | k rS t j
d d | � ^ qS S( s7 `grep vmlinuz /boot/grub2/grub.cfg | sed 's/root=.*//'`R. s utf-8R= Nt vmlinuzs root=.*RM ( R0 R, R1 t GRUB2_CFGR: R2 R3 R> RA RF t sub( R4 RQ RD ( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt collect_grub2_entries s
c C s t t j t � � S( N( t sortedt globt YUM_REPOS_GLOB( ( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt collect_yum_repos s c C sC g t j D] \ } } | � r
| ^ q
} i | d 6t j � d 6S( s8 presence-only port of the kcdoctor.sh detect_cp() probest cpt softaculous( R t CONTROL_PANEL_PROBESt has_softaculous( t namet probet panels( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt collect_control_panel s +c C s_ t d d g d t d t �\ } } } | j | d t j | pB d j d d � d d
��d S( s `ipcs -m | sed -e s/-/=/g`t ipcss -mRK RL R/ RM Rj t =t encodings utf-8N( R RN R+ R RH R= ( R# R RO RP ( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt collect_ipcs% s 'c C s� t j j t � s, | j d j t � � d St t d � �8 } g | j � j � D] } d | k rQ | ^ qQ } Wd QX| j | d d j
| � d �d S( s$ `grep ' install ' /var/log/dpkg.log`s file not found: {0}NR. s install R/ s
( R0 R, R1 t DPKG_LOGR! R" R2 R3 RA R+ t join( R# R R4 RD RC ( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt collect_dpkg_install_log, s 7c C s� t j j d � s$ t j j d � r- d } n^ t j j d � r� d } y t | d � Wq� t k
r� } | j d j | � � q� Xn d } | j | | � d S( Ns /usr/bin/rpms /bin/rpmsF rpm -q -a --queryformat="%{N}|%{V}-%{R}|%{arch}|%{INSTALLTIME:date}\n"s
/usr/bin/dpkgsJ /usr/bin/dpkg-query -W -f "${binary:Package}|${Version}|${Architecture}\n"s dpkg.logs failed to collect dpkg.log: {0}s echo "unknown package manager"( R0 R, R1 R� R R! R" R' ( R# R t packages_cmdR$ ( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt collect_packages9 s $ c
s� t j d � � g } x� t t j t � � D]n } t j j | � r� xS t j | � D]2 \ � } } | j � f d � t | � D� � qS Wq+ | j
| � q+ W| s� | j d j t � � d Sg } x� | D]� } y( t
| d � � } | j � } Wd QXWn/ t k
r)} | j d j | | � � q� n Xt j d j | � d d �� | j � � f d
� | j � D� � q� W| j | d d j | � d �d S(
s/ `grep -rE '^(deb|URIs)' /etc/apt/sources.list*`s ^(deb|URIs)c 3 s$ | ] } t j j � | � Vq d S( N( R0 R, R� ( t .0Rz ( t root( s, /usr/libexec/kcare/python/kcarectl/doctor.pys <genexpr>U s s file not found: {0}NR. s failed to read {0}: {1}s {0}:R� s utf-8c 3 s( | ] } � j | � r � | Vq d S( N( RB ( R� RD ( t prefixRE ( s, /usr/libexec/kcare/python/kcarectl/doctor.pys <genexpr>j s R/ s
( RF RG Rr Rs t APT_SOURCES_GLOBR0 R, t isdirt walkt extendt appendR! R" R2 R3 R R RH RA R+ R� (
R# R t filesR, RO t namesRC R4 t contentR$ ( ( R� RE R� s, /usr/libexec/kcare/python/kcarectl/doctor.pyt collect_apt_sourcesL s* *
*c C s
t t j t � � } | s5 | j d j t � � d Sg } x� | D]� } y3 t | d � � } | j � j � t } Wd QXWn/ t k
r� } | j d j | | � � qB n Xt
j d j | � d d �} | j | d j
| � � qB W| j | d d j
| � d �d S(
s% `tail -n10000 /var/log/libcare/*.log`s file not found: {0}NR. s failed to read {0}: {1}s ==> {0} <==
R� s utf-8s
R/ ( Rr Rs t LIBCARE_LOGS_GLOBR! R" R2 R3 RA t LOG_TAIL_LINESR R RH R� R� R+ ( R# R t pathst chunksR, R4 RC R$ t header( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt collect_libcare_logso s
! c s� t j j t j � s d S| j | d j t j � � x� t j t j � D]� \ } } } x| t | � D]n � t � f d � t
D� � rd t j j | � � } t j j | t j � } | j
d j | � d | �qd qd WqH Wd S( s3 crashreporter artifacts from /var/cache/kcare/dumpsNs
ls -lR {0}c 3 s! | ] } t j � | � Vq d S( N( t fnmatch( R� RI ( Rz ( s, /usr/libexec/kcare/python/kcarectl/doctor.pys <genexpr>� s s crashreporter/{0}R* ( R0 R, R� R t
KDUMPS_DIRR' R" R� Rr t anyt CRASH_DUMP_PATTERNSR� t relpathR+ ( R# R R� RO R� R, t rel_path( ( Rz s, /usr/libexec/kcare/python/kcarectl/doctor.pyt collect_crash_dumps� s " c C s� t j j t � s d St j � } t j j | � s8 d S| j | d j | � � x{ t t
j
t j j | d � � � D]U } t j j | � s� qv n | j d j t j j
| � � d t j j | d � �qv Wd S( Ns
ls -lR {0}t *s kdump/{0}/vmcore-dmesg.txtR* s vmcore-dmesg.txt( R0 R, R1 t
KDUMP_CONFR t get_kdump_rootR� R' R" Rr Rs R� R+ t basename( R# R t
kdump_roott evt( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt
collect_kdump� s +c s g } x� t j t � D]� \ } } } | j t � rA g | (q n x� t | � D]� � t � f d � t D� � rN t j j | � � } y( t
| d � � } | j � } Wd QXWn t k
r� d } n X| j
t j d j | � � | d � qN qN Wq W| j | d d j | � �d S( sG tunables gathered by the kcdoctor.sh get_sysfs_info() `find /sys` callsc 3 s! | ] } t j � | � Vq d S( N( R� ( R� RI ( Rz ( s, /usr/libexec/kcare/python/kcarectl/doctor.pys <genexpr>� s R. NRM s cat {0}
s
R/ ( R0 R� t
SYSFS_ROOTt
startswitht SYSFS_EXCLUDE_PREFIXESRr R� t SYSFS_NAME_PATTERNSR, R� R2 R3 R R� R RH R" R+ ( R# R R� R� t dirsR� R, R4 R� ( ( Rz s, /usr/libexec/kcare/python/kcarectl/doctor.pyt collect_sysfs_config� s
2t doctor_versiont virt_whats /usr/libexec/kcare/virt-whatRS t main_ipR` t kernel_idt dateRi t redhat_releases /etc/redhat-releaset debian_versions /etc/debian_versiont
os_releases /etc/os-releaset issues
/etc/issuet sysconfig_kernels /etc/sysconfig/kernelt proc_uptimes /proc/uptimet proc_loadavgs
/proc/loadavgt proc_cmdlines
/proc/cmdlinet proc_versions
/proc/versiont default_grubs /etc/default/grubt DEFAULTt
grub2_entriest sshd_ports /etc/ssh/sshd_configt Portt yum_repos_dt
control_panelt syslogs tail -n{0} /var/log/syslogt dmesgt messagess tail -n{0} /var/log/messagest ls_var_cache_kcares ls -lR /var/cache/kcare/s
kcare.confs /etc/sysconfig/kcare/kcare.conft cpuinfos cat /proc/cpuinfot proc_vmstats /proc/vmstatt proc_devicess
/proc/devicest proc_diskstatss /proc/diskstatst proc_mdstats /proc/mdstatt proc_meminfos
/proc/meminfot
proc_swapss /proc/swapst proc_filesystemss /proc/filesystemst proc_mountss /proc/mountst proc_interruptss /proc/interruptss grub.confs /boot/grub/grub.conft proc_moduless
/proc/moduless grub2.cfgt
proc_zoneinfos /proc/zoneinfot ls_boot_configss: ls /etc/grub.conf /boot/grub/grub.conf /boot/grub/menu.lstt ls_boots ls -l /boott printenvt dmidecodeR~ t sysctls sysctl -as sysctl.confs /etc/sysctl.confs
packages.listt lspcis lspci -vvt dpkg_ls dpkg -lt apt_sourcess yum.confs
/etc/yum.conft yum_repolists yum repolistt libcare_logss kcarectl.logs tail -n{0} /var/log/kcarectl.logs
kdump.confs
kcare-crons /etc/cron.d/kcare-crons crashreporter/lss kdump/lst aa_statuss aa-statust sestatust kprobes_lists" cat /sys/kernel/debug/kprobes/listt sysfs_configt lsblks lsblk -ft dfs df -ht loggerc C so t � } | �Z i } x t D] } | j | | � q W| j d | � x t D] } | j | � qN WWd QX| S( Ns doctor.json( R t FIELD_SPECSR% t add_jsont
FILE_SPECS( R# R; t fieldt spec( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt prepare_doctor_report s
c C sB | s t j rH | r d n d } t j d j | � d t �| � d St j � d k rx t j d d t �| � d St � } | d k r� t j
d � | � d Sz� yB | j � } t j d j | � d t �t j
d j | � � WnB t t t j t j f k
r.} t j
d
j | � � | � n XWd | j � Xd S( s� Collect and upload the doctor report via the generic uploads proxy.
PUTs the archive to ``/upload/kcarectl-doctor/`` -- the same modern
pipeline as ``kernel-anomaly`` -- which ePortal proxies transparently
to the patch server, so no ePortal change is required. Any 2xx reply
is success, including the ``REDUCED_REPORT`` ``204`` where the proxy
intentionally drops the report (``send()`` returns without raising, so
we do not fall back).
Falls back to the legacy ``kcdoctor.sh`` flow (``fallback``) when:
* ``force_fallback`` (the ``--doctor --fallback`` CLI flag) or
``config.FORCE_DOCTOR_FALLBACK`` is set -- the rollout kill-switch.
``--fallback`` is the one-off, per-invocation form; the config knob
is settable per host in kcare.conf or pushed fleet-wide via the
``KC-Flag-Force-Doctor-Fallback`` feature flag. Either forces the
legacy flow on every host, registered or not, before the new path
is even considered;
* the host is not registered, i.e. there is no auth string (OQ-4: the
proxy upload requires auth, so the new path is not even attempted);
* report collection failed (``prepare_doctor_report`` returned None);
* the new route is unavailable on an older patch server / ePortal --
a 404 / 405 / 403, surfaced as ``HTTPError``, or a connection-level
failure. ``send()`` drives ``http_utils.upload_file`` (raw
``httplib``, not ``urllib``), so transport failures surface as an
``OSError`` subclass -- ``socket.timeout``, ``ConnectionRefused``,
``socket.gaierror`` (DNS), ``ssl.SSLError`` -- or an
``httplib.HTTPException``, *not* ``URLError``; all of these mean the
new path is unusable. ``upload_file`` raises only on status >= 400,
so a redirect is not treated as a failure; the proxy and patch
server never redirect uploads.
:param fallback: legacy ``kcdoctor.sh`` runner, injected by the caller
(KPT-6065) to keep this module free of an ``__init__`` import cycle.
:param force_fallback: when True (``--doctor --fallback``), skip the v2
path and run ``fallback`` directly; the per-invocation form of
``config.FORCE_DOCTOR_FALLBACK``.
s --fallback requesteds FORCE_DOCTOR_FALLBACK is sets. doctor: {0}; using the legacy kcdoctor.sh flowt print_msgNsA doctor: host is not registered; using the legacy kcdoctor.sh flowsC doctor: report collection failed; using the legacy kcdoctor.sh flows doctor: report uploaded as {0}s, Please provide this filename to support: {0}sJ doctor: report upload unavailable ({0}); using the legacy kcdoctor.sh flow( R t FORCE_DOCTOR_FALLBACKR t loginfoR" t FalseR t get_http_auth_stringR: R� t logwarnt sendR t
print_wrapperR
R R t
HTTPExceptiont sockett errort remove_archive( t fallbackt force_fallbackt srcR# t upload_namet err( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt send_doctor_report- s6 )
!( s enable*s nr_*s max_*s *cnt*( s /sys/kernel/debug/tracing/eventss /sys/kernel/tracing/eventss /sys/devices( s
[0-9]*.logs kmsg*log(N R& R� Rs R0 RF R� t kcarectlR R R R R R R R R t kcarectl.process_utilsR t
kcarectl.py23R
R R R� t typingR
R R R R RU Ro R� R� R� Rt R� R� R� R� R� R� t DataPackageR t objectR R) R- R5 R7 R8 R@ RJ RT RV t cachedR] Ra Rc Rm Rq Ru R} R� R� R� R� R� R� R� R� RN R� R" R� t catch_errorsR� R� R� ( ( ( s, /usr/libexec/kcare/python/kcarectl/doctor.pyt <module> s� @ +
#