How to find XML API requests sent to VA Agent

Question

I have problem with Virtuozzo Automator, how do I find out which exact XML API command it failed to process?

Answer

1. Find the UUID of the VM or container in question with prlctl list:

# prlctl list
UUID                                   STATUS    IP_ADDR      T   NAME
{fc2247dd-c22a-4c4b-b1b4-897d8a15462d} running   10.0.102.20  CT  va-mn

Note: for the legacy containers migrated from the earlier versions of Virtuozzo products the UUID may not match the EID. As an additional step for such containers re-check the EID instead of the UUID. Find the EID of the VE in the container's private area in .vza/eid.conf file:

# cat /vz/private/701/.vza/eid.conf
cee8df85-bab4-a94e-9799-4fd8f8fb7a96

2. Enable debug logging for VA Agent and restart its service, e.g:

# sed '/log_level/s/>.*.</>5</g' /var/opt/pva/agent/etc/vzagent.conf -i

# grep log_level /var/opt/pva/agent/etc/vzagent.conf
<log_level>5</log_level>
# pvaagent restart

NOTE: vzagent.conf may be located inside /vz/pva/agent/etc directory

3. Login to the PVA MN (or PowerPanel 1.0 if you are using Virtuozzo Automation 6.x), prepare to reproduce the problem. This step is needed to avoid extra logs getting to the output in the next step.

4. It is suggested to attach the tail -f command to the log file, right before any actions in the GUI, to capture only the required part of the logs:

# tail -f /var/log/pva/agent/2021.05.03-vzagent.log > /tmp/xmlapi.701.log

5. Reproduce the problem in GUI

6. Detach the tail from the log by pressing Ctrl-C.

NOTE: Do not forget to decrease the log level back to the original value once you have finished troubleshooting. Having a log level higher than 3 may cause rapid growth of log files, possibly filling up all free space on the disk and leading to data corruption and downtime

# sed '/log_level/s/>.*.</>3</g' /var/opt/pva/agent/etc/vzagent.conf -i

# pvaagent restart

7. Find the packet by operator name (res_log) or by the container EID found earlier. Below is an example of the exact packet sent by Power Panel 1.0 to VA Agent to retrieve all traffic statistics for container 701:

# less /tmp/xmlapi.701.log T=2011.05.03 16:05:40.394; L=(debug) ; PID=8615; TID=1187964016; [VZLDirectorPrototype::handleIncoming] Incoming message
<packet xmlns:ns1="http://www.swsoft.com/webservices/vzl/4.0.0/types"" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"" uid="" priority="0" time="2011-05-03T09:05:40+0000" type="2" id="33kgfkyba
aaaasvy6ftc1qqcciefdaaa" version="4.5.0">
<target>res_log</target>
<dst>
<host>bd76f8ec-f5c8-3b4f-a558-551bc60ed939</host>
</dst>
<data>
<res_log>
<get_log>
<eid>cee8df85-bab4-a94e-9799-4fd8f8fb7a96</eid>
<class>
<name>counters_vz_net</name>
</class>
<start_time>2011-05-01T00:00:00+0000</start_time>
<end_time>2011-06-01T00:00:00+0000</end_time>
<period>86400</period>
<report_empty/>
</get_log>
</res_log>
</data>
<origin>vzclient2-bd76f8ec-f5c8-3b4f-a558-551bc60ed939</origin>
<cookie>
<src>
<index>3</index>
<director>pvappd</director>
<target>webcp</target>
</src>
</cookie>
<auth xsi:type="ns1:tokenType">
<ns1:user>AQUAAAAAIAEOWHmuxkFDkp51B+UXSMzRAAAAAA==</ns1:user>
<ns1:groups>
<ns1:sid>AQEAAAAAA+jpAwAA</ns1:sid>
<ns1:sid>AQUAAAAAIAAOWHmuxkFDkp51B+UXSMzRAAAAAA==</ns1:sid>
<ns1:sid>AQUAAAAAIADs+Ha9yPVPO6VYVRvGDtk5AAAAAA==</ns1:sid>
<ns1:sid>AQUAAAAAIAEOWHmuxkFDkp51B+UXSMzRAAAAAA==</ns1:sid>
</ns1:groups>
<ns1:deny_only_sids/>
<ns1:privileges/>
<ns1:name xsi:type="ns1:auth_nameType">
<ns1:domain>YWU3OTU4MGUtNDFjNi05MjQzLTllNzUtMDdlNTE3NDhjY2Qx</ns1:domain>
<ns1:name>cm9vdA==</ns1:name>
<ns1:realm>00000000-0000-0000-0100-000000000000</ns1:realm>
</ns1:name>
</auth>
</packet>

Now, using the schema from page 459 of the XML API reference, we can cut off unnecessary fields, like cookie and auth, and compose our own request:

<packet version="4.5.0">
<target>res_log</target>
<data>
<res_log>
<get_log>
<eid>cee8df85-bab4-a94e-9799-4fd8f8fb7a96</eid>
<class>
<name>counters_vz_net</name>
</class>
<start_time>2011-05-01T00:00:00+0000</start_time>
<end_time>2011-06-01T00:00:00+0000</end_time>
<period>86400</period>
<report_empty/>
</get_log>
</res_log>
</data>
</packet>

To verify this is a working request, we can send it as described in this article.

VA Control Center packets

Packets also may be captured from va-cc log, as it contains complete packets without truncating them.

To capture packets in va-cc log:

1. Increase the log level, see the original vs. increased log level:

   # diff /etc/opt/pva/cc/pvacc.conf.orig /etc/opt/pva/cc/pvacc.conf
   34,35c34,35
   <               <level>1</level>
   <               <agent_level>3</agent_level>
   
   ---
   
   >               <level>50</level>
   >               <agent_level>5</agent_level>
   

2. Restart va-cc.service (it will not interrupt active operations like backup or migration, but all active sessions will be dropped):

   # systemctl restart va-cc.service
   

3. Go to VA MN GUI and perform actions you want to perform with API query.
4. Open /var/log/pva/cc/agent.log and find the required packet.

Additional information

Virtuozzo Automator quick troubleshooting

How to send an XML API request to Virtual Automation (VA)