fun_args not passed properly via salt-syndic

[ze42]

Description of Issue/Question

In return events (salt/job/<jid>/ret/<minion>), the fun_args get striped from when returned by the syndic.

Setup

• Two servers.
• Both having a local salt-master and a local salt-minion connected to it.
• Second server having a salt-syndic to report to the first.

In examples bellow: d-ze-std01-empty is the main master, d-ze-std04-empty is the one with a syndic on it.

Steps to Reproduce Issue

Just lanch a command on the syndic'ed minion... and watch for the events on both salt-master.

Launching the command:

d-ze-std01-empty# salt '*' test.ping

Watching the results, from the main master:

# salt-run state.event 'salt/job/*/ret/d-ze*' | while read -r tag data ; do
  echo $tag ;
  echo $data | jq --color-output . ;
done

{
  "fun_args": [],
  "jid": "20180201152150926020",
  "return": true,
  "retcode": 0,
  "success": true,
  "cmd": "_return",
  "_stamp": "2018-02-01T15:21:50.993334",
  "fun": "test.ping",
  "id": "d-ze-std01-empty"
}
salt/job/20180201152150926020/ret/d-ze-std04-empty
{
  "fun_args": null,
  "jid": "20180201152150926020",
  "return": true,
  "retcode": 0,
  "success": true,
  "_stamp": "2018-02-01T15:21:51.476612",
  "fun": "test.ping",
  "id": "d-ze-std04-empty"
}

From the secondary master:

salt/job/20180201152150926020/ret/d-ze-std04-empty
{
  "fun_args": [],
  "jid": "20180201152150926020",
  "return": true,
  "retcode": 0,
  "success": true,
  "cmd": "_return",
  "_stamp": "2018-02-01T15:21:51.007583",
  "fun": "test.ping",
  "id": "d-ze-std04-empty"
}

Notice how "fun_args" changes from [] to null when seen on the main master...

(Also tested with test.args having bigger data... same results, appears properly on the master connected to the minion, stripped once moving up via salt-syndic).

Versions Report

Tested on 2016.11.6+ds-1 and 2017.7.2+ds-1 (same version on both server).

Latest tested:

# salt --versions-report
Salt Version:
           Salt: 2017.7.2
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Nov 24 2017, 17:33:09)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1
 
System Versions:
           dist: debian 9.3 
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 4.4.93-cerberhost-0.13.0deb8+nbs20171018-grsec
         system: Linux
        version: debian 9.3 

[gtmanfred]

I have verified this.

Thanks for reporting,
Daniel

[ze42]

Any idea when someone might work on it?

[gtmanfred]

@saltstack/team-triage this one might make sense as a blocker

@ze42 if it is not a blocked for a release, then I have no idea when it will be worked on, but a pull request to fix it would be greatly appreciated

[ze42]

If I had enough knowledge about the workflow and where to fix it, there would already be a PR.

[sagetherage]

For 2 release cycles we have not been able to write a test for this, so I am putting in back into Approved until tests are written to determine the release cycle.

[keslerm]

Just got hit with this bug too which was breaking some of our tooling in our migration to Syndic

[keslerm]

Doing some quick digging I believe the issue is here https://github.com/saltstack/salt/blob/master/salt/master.py#L1751-L1752

                if "arg" in load:
                    ret["fun_args"] = load["arg"]

[keslerm]

There's actually a second part that needs changed as well which is the syndic code in the minion which needs to forward fun_args as well

salt/salt/minion.py

Line 3685 in 9012911

for key in "return", "retcode", "success":

[keslerm]

I've posted and tested a fix here #62809

It still I think needs unit tests however