Skip to content

M
mycard-deploy
Commit 985463fd authored by nanahira's avatar nanahira
Browse files
Options

mitogen

parent 3fda26b0
Hide whitespace changes
Inline Side-by-side
Showing with 2748 additions and 1617 deletions
ansible.cfg
View file @ 985463fd
[defaults]
inventory = inventory.yaml
host_key_checking = False
strategy_plugins = mitogen-0.2.9/ansible_mitogen/plugins/strategy
strategy_plugins = mitogen-0.3.7/ansible_mitogen/plugins/strategy
strategy = mitogen_linear
inventory.example.yaml
View file @ 985463fd
mycard:
hosts:
example.com:
153.34.0.137:
children:
router_nextgen:
rcloud:
vars: # 一般可以什么都不改，想改的话改什么抄什么
ansible_ssh_user: root
ansible_ssh_user: frog
ansible_python_interpreter: python3
authorized_keys: |
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCf7d3z1zJ3+AekdFlF+7BTL8k1X5YrC/Mu23aBBETYxl7N37XM/Ts1u15Xu9l0rgKOe2YphAzexsTmvnBW+3ZJUaJ4gIelw2FWYSHmPkcTtLRrqmSqqCVqyxLrlkvDV1mJmRTQh9rFT3OM2jge4HkN2PSnxuT4AuRwblHcGj2LWD/4XijmHf/BK6GaKownt702ARX18D7d2hYZwO7TgVtJsZclIg3g9SjLgtx6sx+Khlx2/p3HdMo7lYmbucnk1Px11Z0Uq6H7gdUyZhRMWgHnqpnzTy02onhsDhPotUbJdEHJKMZW+eg3X1KiPH143jDHPW/EQjhnPfMBBqaPof1l nanahira@nanahira-another
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAtRix6NrCSXMNpL9WuD6DA198aGirvb8cYIcx5fS98/EWqA8n8yjBEjfLkWZviSh8J6hDw5x4rlZWa777eP+qFfwZO5MjQp/n3cgpZgnbJFRUROuNEyaGQvv09uO05cgRKemVDysqte6xjH6YOts/+oX6dC/JK+Cwi7K0kUETQ2WLLTghyQfLkwKoXkP30v/j18yfyswyWsM1E70stmezMRYswsAeOP6j5/dZiSY9vPCPHJ0w3cGhV+YZcWVE3687cQyf++Iv4AGBzRWlGStGHfb3UB8fkeIClChkQDjjzrxfbrmeS3kC5w6hkbZFsreM8ZvWhDvB1eBxjU9KKbV0iQ== zh99998@gmail.com
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCq9xZpQkZ7MAfRRI2pVHzpCgjWFsGjBDW4ue4d83FtCYYRvZ8MCmRq3kIASpnQUKHWkHR8OZdYVjlCZyVxbVQgF4/YpPzSo64wtWR2gdKEtYLzEgsu8Y6u5hcqPAEFK1bZA01oxj8/sn+LsQOFfhaMfBEx2wplKz8CzgJImzMHX4Zf1VkeNdKvfnSV1oX8DpSMm7eNdg0W84LXi7dIACNi9FU2b/I5Z+OZmjK6eHVQjT4aaRs9fY7Tx9QknDq0i7yhPQQGNTHCA8EQmnYN1pj9jqudNguqZ/PnqGMp2OMSdlBe5XWmERK0mjQPq80sN2bQjIXUKC8XnTtNrxf+GgJVWTCogVdLvRn4C4j32yMDFrtfK1kxLNhZANZGFHDMiv5aRYmeIM3t02DoClt/IvHNB1gOXd0ljJZy9/TExYuI/y7XkC9PWxvkNtl8jEWN6/uyWCQ2kuos92b51BXHynijaKp7qFqj6ae7T4dnKhSVwCJOFR+d8RrMEwDPAMfBveE= frogeater@LAPTOP-5570BFO6
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAtRix6NrCSXMNpL9WuD6DA198aGirvb8cYIcx5fS98/EWqA8n8yjBEjfLkWZviSh8J6hDw5x4rlZWa777eP+qFfwZO5MjQp/n3cgpZgnbJFRUROuNEyaGQvv09uO05cgRKemVDysqte6xjH6YOts/+oX6dC/JK+Cwi7K0kUETQ2WLLTghyQfLkwKoXkP30v/j18yfyswyWsM1E70stmezMRYswsAeOP6j5/dZiSY9vPCPHJ0w3cGhV+YZcWVE3687cQyf++Iv4AGBzRWlGStGHfb3UB8fkeIClChkQDjjzrxfbrmeS3kC5w6hkbZFsreM8ZvWhDvB1eBxjU9KKbV0iQ==
allow_password: false
hypervisor: false
mirror_debian: http://deb.debian.org
......@@ -17,12 +17,12 @@ mycard:
mirror_ubuntu: http://archive.ubuntu.com
mirror_ubuntu_security: http://security.ubuntu.com
mirror_docker: https://download.docker.com
china_mirror: false
china_mirror: true
install_docker: true
upgrade: true
reboot: true
pip_executable_install: pip
registry: https://registry.mycard.moe
registry: null
china_mirror_docker: false
china_mirror_pip: false
china_mirror_omf: false
......
mitogen-0.2.9/README.md deleted 100644 → 0
View file @ 3fda26b0
# Mitogen
<!-- [![Build Status](https://travis-ci.org/dw/mitogen.png?branch=master)](https://travis-ci.org/dw/mitogen}) -->
<a href="https://mitogen.networkgenomics.com/">Please see the documentation</a>.
![](https://i.imgur.com/eBM6LhJ.gif)
[![Total alerts](https://img.shields.io/lgtm/alerts/g/dw/mitogen.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/dw/mitogen/alerts/)
[![Build Status](https://travis-ci.org/dw/mitogen.svg?branch=master)](https://travis-ci.org/dw/mitogen)
[![Pipelines Status](https://dev.azure.com/dw-mitogen/Mitogen/_apis/build/status/dw.mitogen?branchName=master)](https://dev.azure.com/dw-mitogen/Mitogen/_build/latest?definitionId=1?branchName=master)
mitogen-0.2.9/ansible_mitogen/compat/simplejson/__init__.py deleted 100644 → 0
View file @ 3fda26b0
r"""JSON (JavaScript Object Notation) <http://json.org> is a subset of
JavaScript syntax (ECMA-262 3rd edition) used as a lightweight data
interchange format.
:mod:`simplejson` exposes an API familiar to users of the standard library
:mod:`marshal` and :mod:`pickle` modules. It is the externally maintained
version of the :mod:`json` library contained in Python 2.6, but maintains
compatibility with Python 2.4 and Python 2.5 and (currently) has
significant performance advantages, even without using the optional C
extension for speedups.
Encoding basic Python object hierarchies::
>>> import simplejson as json
>>> json.dumps(['foo', {'bar': ('baz', None, 1.0, 2)}])
'["foo", {"bar": ["baz", null, 1.0, 2]}]'
>>> print json.dumps("\"foo\bar")
"\"foo\bar"
>>> print json.dumps(u'\u1234')
"\u1234"
>>> print json.dumps('\\')
"\\"
>>> print json.dumps({"c": 0, "b": 0, "a": 0}, sort_keys=True)
{"a": 0, "b": 0, "c": 0}
>>> from StringIO import StringIO
>>> io = StringIO()
>>> json.dump(['streaming API'], io)
>>> io.getvalue()
'["streaming API"]'
Compact encoding::
>>> import simplejson as json
>>> json.dumps([1,2,3,{'4': 5, '6': 7}], separators=(',',':'))
'[1,2,3,{"4":5,"6":7}]'
Pretty printing::
>>> import simplejson as json
>>> s = json.dumps({'4': 5, '6': 7}, sort_keys=True, indent=4)
>>> print '\n'.join([l.rstrip() for l in s.splitlines()])
{
"4": 5,
"6": 7
}
Decoding JSON::
>>> import simplejson as json
>>> obj = [u'foo', {u'bar': [u'baz', None, 1.0, 2]}]
>>> json.loads('["foo", {"bar":["baz", null, 1.0, 2]}]') == obj
True
>>> json.loads('"\\"foo\\bar"') == u'"foo\x08ar'
True
>>> from StringIO import StringIO
>>> io = StringIO('["streaming API"]')
>>> json.load(io)[0] == 'streaming API'
True
Specializing JSON object decoding::
>>> import simplejson as json
>>> def as_complex(dct):
... if '__complex__' in dct:
... return complex(dct['real'], dct['imag'])
... return dct
...
>>> json.loads('{"__complex__": true, "real": 1, "imag": 2}',
... object_hook=as_complex)
(1+2j)
>>> import decimal
>>> json.loads('1.1', parse_float=decimal.Decimal) == decimal.Decimal('1.1')
True
Specializing JSON object encoding::
>>> import simplejson as json
>>> def encode_complex(obj):
... if isinstance(obj, complex):
... return [obj.real, obj.imag]
... raise TypeError(repr(o) + " is not JSON serializable")
...
>>> json.dumps(2 + 1j, default=encode_complex)
'[2.0, 1.0]'
>>> json.JSONEncoder(default=encode_complex).encode(2 + 1j)
'[2.0, 1.0]'
>>> ''.join(json.JSONEncoder(default=encode_complex).iterencode(2 + 1j))
'[2.0, 1.0]'
Using simplejson.tool from the shell to validate and pretty-print::
$ echo '{"json":"obj"}' | python -m simplejson.tool
{
"json": "obj"
}
$ echo '{ 1.2:3.4}' | python -m simplejson.tool
Expecting property name: line 1 column 2 (char 2)
"""
__version__ = '2.0.9'
__all__ = [
'dump', 'dumps', 'load', 'loads',
'JSONDecoder', 'JSONEncoder',
]
__author__ = 'Bob Ippolito <bob@redivi.com>'
from decoder import JSONDecoder
from encoder import JSONEncoder
_default_encoder = JSONEncoder(
skipkeys=False,
ensure_ascii=True,
check_circular=True,
allow_nan=True,
indent=None,
separators=None,
encoding='utf-8',
default=None,
)
def dump(obj, fp, skipkeys=False, ensure_ascii=True, check_circular=True,
allow_nan=True, cls=None, indent=None, separators=None,
encoding='utf-8', default=None, **kw):
"""Serialize ``obj`` as a JSON formatted stream to ``fp`` (a
``.write()``-supporting file-like object).
If ``skipkeys`` is true then ``dict`` keys that are not basic types
(``str``, ``unicode``, ``int``, ``long``, ``float``, ``bool``, ``None``)
will be skipped instead of raising a ``TypeError``.
If ``ensure_ascii`` is false, then the some chunks written to ``fp``
may be ``unicode`` instances, subject to normal Python ``str`` to
``unicode`` coercion rules. Unless ``fp.write()`` explicitly
understands ``unicode`` (as in ``codecs.getwriter()``) this is likely
to cause an error.
If ``check_circular`` is false, then the circular reference check
for container types will be skipped and a circular reference will
result in an ``OverflowError`` (or worse).
If ``allow_nan`` is false, then it will be a ``ValueError`` to
serialize out of range ``float`` values (``nan``, ``inf``, ``-inf``)
in strict compliance of the JSON specification, instead of using the
JavaScript equivalents (``NaN``, ``Infinity``, ``-Infinity``).
If ``indent`` is a non-negative integer, then JSON array elements and object
members will be pretty-printed with that indent level. An indent level
of 0 will only insert newlines. ``None`` is the most compact representation.
If ``separators`` is an ``(item_separator, dict_separator)`` tuple
then it will be used instead of the default ``(', ', ': ')`` separators.
``(',', ':')`` is the most compact JSON representation.
``encoding`` is the character encoding for str instances, default is UTF-8.
``default(obj)`` is a function that should return a serializable version
of obj or raise TypeError. The default simply raises TypeError.
To use a custom ``JSONEncoder`` subclass (e.g. one that overrides the
``.default()`` method to serialize additional types), specify it with
the ``cls`` kwarg.
"""
# cached encoder
if (not skipkeys and ensure_ascii and
check_circular and allow_nan and
cls is None and indent is None and separators is None and
encoding == 'utf-8' and default is None and not kw):
iterable = _default_encoder.iterencode(obj)
else:
if cls is None:
cls = JSONEncoder
iterable = cls(skipkeys=skipkeys, ensure_ascii=ensure_ascii,
check_circular=check_circular, allow_nan=allow_nan, indent=indent,
separators=separators, encoding=encoding,
default=default, **kw).iterencode(obj)
# could accelerate with writelines in some versions of Python, at
# a debuggability cost
for chunk in iterable:
fp.write(chunk)
def dumps(obj, skipkeys=False, ensure_ascii=True, check_circular=True,
allow_nan=True, cls=None, indent=None, separators=None,
encoding='utf-8', default=None, **kw):
"""Serialize ``obj`` to a JSON formatted ``str``.
If ``skipkeys`` is false then ``dict`` keys that are not basic types
(``str``, ``unicode``, ``int``, ``long``, ``float``, ``bool``, ``None``)
will be skipped instead of raising a ``TypeError``.
If ``ensure_ascii`` is false, then the return value will be a
``unicode`` instance subject to normal Python ``str`` to ``unicode``
coercion rules instead of being escaped to an ASCII ``str``.
If ``check_circular`` is false, then the circular reference check
for container types will be skipped and a circular reference will
result in an ``OverflowError`` (or worse).
If ``allow_nan`` is false, then it will be a ``ValueError`` to
serialize out of range ``float`` values (``nan``, ``inf``, ``-inf``) in
strict compliance of the JSON specification, instead of using the
JavaScript equivalents (``NaN``, ``Infinity``, ``-Infinity``).
If ``indent`` is a non-negative integer, then JSON array elements and
object members will be pretty-printed with that indent level. An indent
level of 0 will only insert newlines. ``None`` is the most compact
representation.
If ``separators`` is an ``(item_separator, dict_separator)`` tuple
then it will be used instead of the default ``(', ', ': ')`` separators.
``(',', ':')`` is the most compact JSON representation.
``encoding`` is the character encoding for str instances, default is UTF-8.
``default(obj)`` is a function that should return a serializable version
of obj or raise TypeError. The default simply raises TypeError.
To use a custom ``JSONEncoder`` subclass (e.g. one that overrides the
``.default()`` method to serialize additional types), specify it with
the ``cls`` kwarg.
"""
# cached encoder
if (not skipkeys and ensure_ascii and
check_circular and allow_nan and
cls is None and indent is None and separators is None and
encoding == 'utf-8' and default is None and not kw):
return _default_encoder.encode(obj)
if cls is None:
cls = JSONEncoder
return cls(
skipkeys=skipkeys, ensure_ascii=ensure_ascii,
check_circular=check_circular, allow_nan=allow_nan, indent=indent,
separators=separators, encoding=encoding, default=default,
**kw).encode(obj)
_default_decoder = JSONDecoder(encoding=None, object_hook=None)
def load(fp, encoding=None, cls=None, object_hook=None, parse_float=None,
parse_int=None, parse_constant=None, **kw):
"""Deserialize ``fp`` (a ``.read()``-supporting file-like object containing
a JSON document) to a Python object.
If the contents of ``fp`` is encoded with an ASCII based encoding other
than utf-8 (e.g. latin-1), then an appropriate ``encoding`` name must
be specified. Encodings that are not ASCII based (such as UCS-2) are
not allowed, and should be wrapped with
``codecs.getreader(fp)(encoding)``, or simply decoded to a ``unicode``
object and passed to ``loads()``
``object_hook`` is an optional function that will be called with the
result of any object literal decode (a ``dict``). The return value of
``object_hook`` will be used instead of the ``dict``. This feature
can be used to implement custom decoders (e.g. JSON-RPC class hinting).
To use a custom ``JSONDecoder`` subclass, specify it with the ``cls``
kwarg.
"""
return loads(fp.read(),
encoding=encoding, cls=cls, object_hook=object_hook,
parse_float=parse_float, parse_int=parse_int,
parse_constant=parse_constant, **kw)
def loads(s, encoding=None, cls=None, object_hook=None, parse_float=None,
parse_int=None, parse_constant=None, **kw):
"""Deserialize ``s`` (a ``str`` or ``unicode`` instance containing a JSON
document) to a Python object.
If ``s`` is a ``str`` instance and is encoded with an ASCII based encoding
other than utf-8 (e.g. latin-1) then an appropriate ``encoding`` name
must be specified. Encodings that are not ASCII based (such as UCS-2)
are not allowed and should be decoded to ``unicode`` first.
``object_hook`` is an optional function that will be called with the
result of any object literal decode (a ``dict``). The return value of
``object_hook`` will be used instead of the ``dict``. This feature
can be used to implement custom decoders (e.g. JSON-RPC class hinting).
``parse_float``, if specified, will be called with the string
of every JSON float to be decoded. By default this is equivalent to
float(num_str). This can be used to use another datatype or parser
for JSON floats (e.g. decimal.Decimal).
``parse_int``, if specified, will be called with the string
of every JSON int to be decoded. By default this is equivalent to
int(num_str). This can be used to use another datatype or parser
for JSON integers (e.g. float).
``parse_constant``, if specified, will be called with one of the
following strings: -Infinity, Infinity, NaN, null, true, false.
This can be used to raise an exception if invalid JSON numbers
are encountered.
To use a custom ``JSONDecoder`` subclass, specify it with the ``cls``
kwarg.
"""
if (cls is None and encoding is None and object_hook is None and
parse_int is None and parse_float is None and
parse_constant is None and not kw):
return _default_decoder.decode(s)
if cls is None:
cls = JSONDecoder
if object_hook is not None:
kw['object_hook'] = object_hook
if parse_float is not None:
kw['parse_float'] = parse_float
if parse_int is not None:
kw['parse_int'] = parse_int
if parse_constant is not None:
kw['parse_constant'] = parse_constant
return cls(encoding=encoding, **kw).decode(s)
mitogen-0.2.9/ansible_mitogen/compat/simplejson/decoder.py deleted 100644 → 0
View file @ 3fda26b0
"""Implementation of JSONDecoder
"""
import re
import sys
import struct
from simplejson.scanner import make_scanner
try:
from simplejson._speedups import scanstring as c_scanstring
except ImportError:
c_scanstring = None
__all__ = ['JSONDecoder']
FLAGS = re.VERBOSE | re.MULTILINE | re.DOTALL
def _floatconstants():
_BYTES = '7FF80000000000007FF0000000000000'.decode('hex')
if sys.byteorder != 'big':
_BYTES = _BYTES[:8][::-1] + _BYTES[8:][::-1]
nan, inf = struct.unpack('dd', _BYTES)
return nan, inf, -inf
NaN, PosInf, NegInf = _floatconstants()
def linecol(doc, pos):
lineno = doc.count('\n', 0, pos) + 1
if lineno == 1:
colno = pos
else:
colno = pos - doc.rindex('\n', 0, pos)
return lineno, colno
def errmsg(msg, doc, pos, end=None):
# Note that this function is called from _speedups
lineno, colno = linecol(doc, pos)
if end is None:
#fmt = '{0}: line {1} column {2} (char {3})'
#return fmt.format(msg, lineno, colno, pos)
fmt = '%s: line %d column %d (char %d)'
return fmt % (msg, lineno, colno, pos)
endlineno, endcolno = linecol(doc, end)
#fmt = '{0}: line {1} column {2} - line {3} column {4} (char {5} - {6})'
#return fmt.format(msg, lineno, colno, endlineno, endcolno, pos, end)
fmt = '%s: line %d column %d - line %d column %d (char %d - %d)'
return fmt % (msg, lineno, colno, endlineno, endcolno, pos, end)
_CONSTANTS = {
'-Infinity': NegInf,
'Infinity': PosInf,
'NaN': NaN,
}
STRINGCHUNK = re.compile(r'(.*?)(["\\\x00-\x1f])', FLAGS)
BACKSLASH = {
'"': u'"', '\\': u'\\', '/': u'/',
'b': u'\b', 'f': u'\f', 'n': u'\n', 'r': u'\r', 't': u'\t',
}
DEFAULT_ENCODING = "utf-8"
def py_scanstring(s, end, encoding=None, strict=True, _b=BACKSLASH, _m=STRINGCHUNK.match):
"""Scan the string s for a JSON string. End is the index of the
character in s after the quote that started the JSON string.
Unescapes all valid JSON string escape sequences and raises ValueError
on attempt to decode an invalid string. If strict is False then literal
control characters are allowed in the string.
Returns a tuple of the decoded string and the index of the character in s
after the end quote."""
if encoding is None:
encoding = DEFAULT_ENCODING
chunks = []
_append = chunks.append
begin = end - 1
while 1:
chunk = _m(s, end)
if chunk is None:
raise ValueError(
errmsg("Unterminated string starting at", s, begin))
end = chunk.end()
content, terminator = chunk.groups()
# Content is contains zero or more unescaped string characters
if content:
if not isinstance(content, unicode):
content = unicode(content, encoding)
_append(content)
# Terminator is the end of string, a literal control character,
# or a backslash denoting that an escape sequence follows
if terminator == '"':
break
elif terminator != '\\':
if strict:
msg = "Invalid control character %r at" % (terminator,)
#msg = "Invalid control character {0!r} at".format(terminator)
raise ValueError(errmsg(msg, s, end))
else:
_append(terminator)
continue
try:
esc = s[end]
except IndexError:
raise ValueError(
errmsg("Unterminated string starting at", s, begin))
# If not a unicode escape sequence, must be in the lookup table
if esc != 'u':
try:
char = _b[esc]
except KeyError:
msg = "Invalid \\escape: " + repr(esc)
raise ValueError(errmsg(msg, s, end))
end += 1
else:
# Unicode escape sequence
esc = s[end + 1:end + 5]
next_end = end + 5
if len(esc) != 4:
msg = "Invalid \\uXXXX escape"
raise ValueError(errmsg(msg, s, end))
uni = int(esc, 16)
# Check for surrogate pair on UCS-4 systems
if 0xd800 <= uni <= 0xdbff and sys.maxunicode > 65535:
msg = "Invalid \\uXXXX\\uXXXX surrogate pair"
if not s[end + 5:end + 7] == '\\u':
raise ValueError(errmsg(msg, s, end))
esc2 = s[end + 7:end + 11]
if len(esc2) != 4:
raise ValueError(errmsg(msg, s, end))
uni2 = int(esc2, 16)
uni = 0x10000 + (((uni - 0xd800) << 10) | (uni2 - 0xdc00))
next_end += 6
char = unichr(uni)
end = next_end
# Append the unescaped character
_append(char)
return u''.join(chunks), end
# Use speedup if available
scanstring = c_scanstring or py_scanstring
WHITESPACE = re.compile(r'[ \t\n\r]*', FLAGS)
WHITESPACE_STR = ' \t\n\r'
def JSONObject((s, end), encoding, strict, scan_once, object_hook, _w=WHITESPACE.match, _ws=WHITESPACE_STR):
pairs = {}
# Use a slice to prevent IndexError from being raised, the following
# check will raise a more specific ValueError if the string is empty
nextchar = s[end:end + 1]
# Normally we expect nextchar == '"'
if nextchar != '"':
if nextchar in _ws:
end = _w(s, end).end()
nextchar = s[end:end + 1]
# Trivial empty object
if nextchar == '}':
return pairs, end + 1
elif nextchar != '"':
raise ValueError(errmsg("Expecting property name", s, end))
end += 1
while True:
key, end = scanstring(s, end, encoding, strict)
# To skip some function call overhead we optimize the fast paths where
# the JSON key separator is ": " or just ":".
if s[end:end + 1] != ':':
end = _w(s, end).end()
if s[end:end + 1] != ':':
raise ValueError(errmsg("Expecting : delimiter", s, end))
end += 1
try:
if s[end] in _ws:
end += 1
if s[end] in _ws:
end = _w(s, end + 1).end()
except IndexError:
pass
try:
value, end = scan_once(s, end)
except StopIteration:
raise ValueError(errmsg("Expecting object", s, end))
pairs[key] = value
try:
nextchar = s[end]
if nextchar in _ws:
end = _w(s, end + 1).end()
nextchar = s[end]
except IndexError:
nextchar = ''
end += 1
if nextchar == '}':
break
elif nextchar != ',':
raise ValueError(errmsg("Expecting , delimiter", s, end - 1))
try:
nextchar = s[end]
if nextchar in _ws:
end += 1
nextchar = s[end]
if nextchar in _ws:
end = _w(s, end + 1).end()
nextchar = s[end]
except IndexError:
nextchar = ''
end += 1
if nextchar != '"':
raise ValueError(errmsg("Expecting property name", s, end - 1))
if object_hook is not None:
pairs = object_hook(pairs)
return pairs, end
def JSONArray((s, end), scan_once, _w=WHITESPACE.match, _ws=WHITESPACE_STR):
values = []
nextchar = s[end:end + 1]
if nextchar in _ws:
end = _w(s, end + 1).end()
nextchar = s[end:end + 1]
# Look-ahead for trivial empty array
if nextchar == ']':
return values, end + 1
_append = values.append
while True:
try:
value, end = scan_once(s, end)
except StopIteration:
raise ValueError(errmsg("Expecting object", s, end))
_append(value)
nextchar = s[end:end + 1]
if nextchar in _ws:
end = _w(s, end + 1).end()
nextchar = s[end:end + 1]
end += 1
if nextchar == ']':
break
elif nextchar != ',':
raise ValueError(errmsg("Expecting , delimiter", s, end))
try:
if s[end] in _ws:
end += 1
if s[end] in _ws:
end = _w(s, end + 1).end()
except IndexError:
pass
return values, end
class JSONDecoder(object):
"""Simple JSON <http://json.org> decoder
Performs the following translations in decoding by default:
+---------------+-------------------+
| JSON | Python |
+===============+===================+
| object | dict |
+---------------+-------------------+
| array | list |
+---------------+-------------------+
| string | unicode |
+---------------+-------------------+
| number (int) | int, long |
+---------------+-------------------+
| number (real) | float |
+---------------+-------------------+
| true | True |
+---------------+-------------------+
| false | False |
+---------------+-------------------+
| null | None |
+---------------+-------------------+
It also understands ``NaN``, ``Infinity``, and ``-Infinity`` as
their corresponding ``float`` values, which is outside the JSON spec.
"""
def __init__(self, encoding=None, object_hook=None, parse_float=None,
parse_int=None, parse_constant=None, strict=True):
"""``encoding`` determines the encoding used to interpret any ``str``
objects decoded by this instance (utf-8 by default). It has no
effect when decoding ``unicode`` objects.
Note that currently only encodings that are a superset of ASCII work,
strings of other encodings should be passed in as ``unicode``.
``object_hook``, if specified, will be called with the result
of every JSON object decoded and its return value will be used in
place of the given ``dict``. This can be used to provide custom
deserializations (e.g. to support JSON-RPC class hinting).
``parse_float``, if specified, will be called with the string
of every JSON float to be decoded. By default this is equivalent to
float(num_str). This can be used to use another datatype or parser
for JSON floats (e.g. decimal.Decimal).
``parse_int``, if specified, will be called with the string
of every JSON int to be decoded. By default this is equivalent to
int(num_str). This can be used to use another datatype or parser
for JSON integers (e.g. float).
``parse_constant``, if specified, will be called with one of the
following strings: -Infinity, Infinity, NaN.
This can be used to raise an exception if invalid JSON numbers
are encountered.
"""
self.encoding = encoding
self.object_hook = object_hook
self.parse_float = parse_float or float
self.parse_int = parse_int or int
self.parse_constant = parse_constant or _CONSTANTS.__getitem__
self.strict = strict
self.parse_object = JSONObject
self.parse_array = JSONArray
self.parse_string = scanstring
self.scan_once = make_scanner(self)
def decode(self, s, _w=WHITESPACE.match):
"""Return the Python representation of ``s`` (a ``str`` or ``unicode``
instance containing a JSON document)
"""
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
end = _w(s, end).end()
if end != len(s):
raise ValueError(errmsg("Extra data", s, end, len(s)))
return obj
def raw_decode(self, s, idx=0):
"""Decode a JSON document from ``s`` (a ``str`` or ``unicode`` beginning
with a JSON document) and return a 2-tuple of the Python
representation and the index in ``s`` where the document ended.
This can be used to decode a JSON document from a string that may
have extraneous data at the end.
"""
try:
obj, end = self.scan_once(s, idx)
except StopIteration:
raise ValueError("No JSON object could be decoded")
return obj, end
mitogen-0.2.9/ansible_mitogen/compat/simplejson/encoder.py deleted 100644 → 0
View file @ 3fda26b0
"""Implementation of JSONEncoder
"""
import re
try:
from simplejson._speedups import encode_basestring_ascii as c_encode_basestring_ascii
except ImportError:
c_encode_basestring_ascii = None
try:
from simplejson._speedups import make_encoder as c_make_encoder
except ImportError:
c_make_encoder = None
ESCAPE = re.compile(r'[\x00-\x1f\\"\b\f\n\r\t]')
ESCAPE_ASCII = re.compile(r'([\\"]|[^\ -~])')
HAS_UTF8 = re.compile(r'[\x80-\xff]')
ESCAPE_DCT = {
'\\': '\\\\',
'"': '\\"',
'\b': '\\b',
'\f': '\\f',
'\n': '\\n',
'\r': '\\r',
'\t': '\\t',
}
for i in range(0x20):
#ESCAPE_DCT.setdefault(chr(i), '\\u{0:04x}'.format(i))
ESCAPE_DCT.setdefault(chr(i), '\\u%04x' % (i,))
# Assume this produces an infinity on all machines (probably not guaranteed)
INFINITY = float('1e66666')
FLOAT_REPR = repr
def encode_basestring(s):
"""Return a JSON representation of a Python string
"""
def replace(match):
return ESCAPE_DCT[match.group(0)]
return '"' + ESCAPE.sub(replace, s) + '"'
def py_encode_basestring_ascii(s):
"""Return an ASCII-only JSON representation of a Python string
"""
if isinstance(s, str) and HAS_UTF8.search(s) is not None:
s = s.decode('utf-8')
def replace(match):
s = match.group(0)
try:
return ESCAPE_DCT[s]
except KeyError:
n = ord(s)
if n < 0x10000:
#return '\\u{0:04x}'.format(n)
return '\\u%04x' % (n,)
else:
# surrogate pair
n -= 0x10000
s1 = 0xd800 | ((n >> 10) & 0x3ff)
s2 = 0xdc00 | (n & 0x3ff)
#return '\\u{0:04x}\\u{1:04x}'.format(s1, s2)
return '\\u%04x\\u%04x' % (s1, s2)
return '"' + str(ESCAPE_ASCII.sub(replace, s)) + '"'
encode_basestring_ascii = c_encode_basestring_ascii or py_encode_basestring_ascii
class JSONEncoder(object):
"""Extensible JSON <http://json.org> encoder for Python data structures.
Supports the following objects and types by default:
+-------------------+---------------+
| Python | JSON |
+===================+===============+
| dict | object |
+-------------------+---------------+
| list, tuple | array |
+-------------------+---------------+
| str, unicode | string |
+-------------------+---------------+
| int, long, float | number |
+-------------------+---------------+
| True | true |
+-------------------+---------------+
| False | false |
+-------------------+---------------+
| None | null |
+-------------------+---------------+
To extend this to recognize other objects, subclass and implement a
``.default()`` method with another method that returns a serializable
object for ``o`` if possible, otherwise it should call the superclass
implementation (to raise ``TypeError``).
"""
item_separator = ', '
key_separator = ': '
def __init__(self, skipkeys=False, ensure_ascii=True,
check_circular=True, allow_nan=True, sort_keys=False,
indent=None, separators=None, encoding='utf-8', default=None):
"""Constructor for JSONEncoder, with sensible defaults.
If skipkeys is false, then it is a TypeError to attempt
encoding of keys that are not str, int, long, float or None. If
skipkeys is True, such items are simply skipped.
If ensure_ascii is true, the output is guaranteed to be str
objects with all incoming unicode characters escaped. If
ensure_ascii is false, the output will be unicode object.
If check_circular is true, then lists, dicts, and custom encoded
objects will be checked for circular references during encoding to
prevent an infinite recursion (which would cause an OverflowError).
Otherwise, no such check takes place.
If allow_nan is true, then NaN, Infinity, and -Infinity will be
encoded as such. This behavior is not JSON specification compliant,
but is consistent with most JavaScript based encoders and decoders.
Otherwise, it will be a ValueError to encode such floats.
If sort_keys is true, then the output of dictionaries will be
sorted by key; this is useful for regression tests to ensure
that JSON serializations can be compared on a day-to-day basis.
If indent is a non-negative integer, then JSON array
elements and object members will be pretty-printed with that
indent level. An indent level of 0 will only insert newlines.
None is the most compact representation.
If specified, separators should be a (item_separator, key_separator)
tuple. The default is (', ', ': '). To get the most compact JSON
representation you should specify (',', ':') to eliminate whitespace.
If specified, default is a function that gets called for objects
that can't otherwise be serialized. It should return a JSON encodable
version of the object or raise a ``TypeError``.
If encoding is not None, then all input strings will be
transformed into unicode using that encoding prior to JSON-encoding.
The default is UTF-8.
"""
self.skipkeys = skipkeys
self.ensure_ascii = ensure_ascii
self.check_circular = check_circular
self.allow_nan = allow_nan
self.sort_keys = sort_keys
self.indent = indent
if separators is not None:
self.item_separator, self.key_separator = separators
if default is not None:
self.default = default
self.encoding = encoding
def default(self, o):
"""Implement this method in a subclass such that it returns
a serializable object for ``o``, or calls the base implementation
(to raise a ``TypeError``).
For example, to support arbitrary iterators, you could
implement default like this::
def default(self, o):
try:
iterable = iter(o)
except TypeError:
pass
else:
return list(iterable)
return JSONEncoder.default(self, o)
"""
raise TypeError(repr(o) + " is not JSON serializable")
def encode(self, o):
"""Return a JSON string representation of a Python data structure.
>>> JSONEncoder().encode({"foo": ["bar", "baz"]})
'{"foo": ["bar", "baz"]}'
"""
# This is for extremely simple cases and benchmarks.
if isinstance(o, basestring):
if isinstance(o, str):
_encoding = self.encoding
if (_encoding is not None
and not (_encoding == 'utf-8')):
o = o.decode(_encoding)
if self.ensure_ascii:
return encode_basestring_ascii(o)
else:
return encode_basestring(o)
# This doesn't pass the iterator directly to ''.join() because the
# exceptions aren't as detailed. The list call should be roughly
# equivalent to the PySequence_Fast that ''.join() would do.
chunks = self.iterencode(o, _one_shot=True)
if not isinstance(chunks, (list, tuple)):
chunks = list(chunks)
return ''.join(chunks)
def iterencode(self, o, _one_shot=False):
"""Encode the given object and yield each string
representation as available.
For example::
for chunk in JSONEncoder().iterencode(bigobject):
mysocket.write(chunk)
"""
if self.check_circular:
markers = {}
else:
markers = None
if self.ensure_ascii:
_encoder = encode_basestring_ascii
else:
_encoder = encode_basestring
if self.encoding != 'utf-8':
def _encoder(o, _orig_encoder=_encoder, _encoding=self.encoding):
if isinstance(o, str):
o = o.decode(_encoding)
return _orig_encoder(o)
def floatstr(o, allow_nan=self.allow_nan, _repr=FLOAT_REPR, _inf=INFINITY, _neginf=-INFINITY):
# Check for specials. Note that this type of test is processor- and/or
# platform-specific, so do tests which don't depend on the internals.
if o != o:
text = 'NaN'
elif o == _inf:
text = 'Infinity'
elif o == _neginf:
text = '-Infinity'
else:
return _repr(o)
if not allow_nan:
raise ValueError(
"Out of range float values are not JSON compliant: " +
repr(o))
return text
if _one_shot and c_make_encoder is not None and not self.indent and not self.sort_keys:
_iterencode = c_make_encoder(
markers, self.default, _encoder, self.indent,
self.key_separator, self.item_separator, self.sort_keys,
self.skipkeys, self.allow_nan)
else:
_iterencode = _make_iterencode(
markers, self.default, _encoder, self.indent, floatstr,
self.key_separator, self.item_separator, self.sort_keys,
self.skipkeys, _one_shot)
return _iterencode(o, 0)
def _make_iterencode(markers, _default, _encoder, _indent, _floatstr, _key_separator, _item_separator, _sort_keys, _skipkeys, _one_shot,
## HACK: hand-optimized bytecode; turn globals into locals
False=False,
True=True,
ValueError=ValueError,
basestring=basestring,
dict=dict,
float=float,
id=id,
int=int,
isinstance=isinstance,
list=list,
long=long,
str=str,
tuple=tuple,
):
def _iterencode_list(lst, _current_indent_level):
if not lst:
yield '[]'
return
if markers is not None:
markerid = id(lst)
if markerid in markers:
raise ValueError("Circular reference detected")
markers[markerid] = lst
buf = '['
if _indent is not None:
_current_indent_level += 1
newline_indent = '\n' + (' ' * (_indent * _current_indent_level))
separator = _item_separator + newline_indent
buf += newline_indent
else:
newline_indent = None
separator = _item_separator
first = True
for value in lst:
if first:
first = False
else:
buf = separator
if isinstance(value, basestring):
yield buf + _encoder(value)
elif value is None:
yield buf + 'null'
elif value is True:
yield buf + 'true'
elif value is False:
yield buf + 'false'
elif isinstance(value, (int, long)):
yield buf + str(value)
elif isinstance(value, float):
yield buf + _floatstr(value)
else:
yield buf
if isinstance(value, (list, tuple)):
chunks = _iterencode_list(value, _current_indent_level)
elif isinstance(value, dict):
chunks = _iterencode_dict(value, _current_indent_level)
else:
chunks = _iterencode(value, _current_indent_level)
for chunk in chunks:
yield chunk
if newline_indent is not None:
_current_indent_level -= 1
yield '\n' + (' ' * (_indent * _current_indent_level))
yield ']'
if markers is not None:
del markers[markerid]
def _iterencode_dict(dct, _current_indent_level):
if not dct:
yield '{}'
return
if markers is not None:
markerid = id(dct)
if markerid in markers:
raise ValueError("Circular reference detected")
markers[markerid] = dct
yield '{'
if _indent is not None:
_current_indent_level += 1
newline_indent = '\n' + (' ' * (_indent * _current_indent_level))
item_separator = _item_separator + newline_indent
yield newline_indent
else:
newline_indent = None
item_separator = _item_separator
first = True
if _sort_keys:
items = dct.items()
items.sort(key=lambda kv: kv[0])
else:
items = dct.iteritems()
for key, value in items:
if isinstance(key, basestring):
pass
# JavaScript is weakly typed for these, so it makes sense to
# also allow them. Many encoders seem to do something like this.
elif isinstance(key, float):
key = _floatstr(key)
elif key is True:
key = 'true'
elif key is False:
key = 'false'
elif key is None:
key = 'null'
elif isinstance(key, (int, long)):
key = str(key)
elif _skipkeys:
continue
else:
raise TypeError("key " + repr(key) + " is not a string")
if first:
first = False
else:
yield item_separator
yield _encoder(key)
yield _key_separator
if isinstance(value, basestring):
yield _encoder(value)
elif value is None:
yield 'null'
elif value is True:
yield 'true'
elif value is False:
yield 'false'
elif isinstance(value, (int, long)):
yield str(value)
elif isinstance(value, float):
yield _floatstr(value)
else:
if isinstance(value, (list, tuple)):
chunks = _iterencode_list(value, _current_indent_level)
elif isinstance(value, dict):
chunks = _iterencode_dict(value, _current_indent_level)
else:
chunks = _iterencode(value, _current_indent_level)
for chunk in chunks:
yield chunk
if newline_indent is not None:
_current_indent_level -= 1
yield '\n' + (' ' * (_indent * _current_indent_level))
yield '}'
if markers is not None:
del markers[markerid]
def _iterencode(o, _current_indent_level):
if isinstance(o, basestring):
yield _encoder(o)
elif o is None:
yield 'null'
elif o is True:
yield 'true'
elif o is False:
yield 'false'
elif isinstance(o, (int, long)):
yield str(o)
elif isinstance(o, float):
yield _floatstr(o)
elif isinstance(o, (list, tuple)):
for chunk in _iterencode_list(o, _current_indent_level):
yield chunk
elif isinstance(o, dict):
for chunk in _iterencode_dict(o, _current_indent_level):
yield chunk
else:
if markers is not None:
markerid = id(o)
if markerid in markers:
raise ValueError("Circular reference detected")
markers[markerid] = o
o = _default(o)
for chunk in _iterencode(o, _current_indent_level):
yield chunk
if markers is not None:
del markers[markerid]
return _iterencode
mitogen-0.2.9/ansible_mitogen/compat/simplejson/scanner.py deleted 100644 → 0
View file @ 3fda26b0
"""JSON token scanner
"""
import re
try:
from simplejson._speedups import make_scanner as c_make_scanner
except ImportError:
c_make_scanner = None
__all__ = ['make_scanner']
NUMBER_RE = re.compile(
r'(-?(?:0|[1-9]\d*))(\.\d+)?([eE][-+]?\d+)?',
(re.VERBOSE | re.MULTILINE | re.DOTALL))
def py_make_scanner(context):
parse_object = context.parse_object
parse_array = context.parse_array
parse_string = context.parse_string
match_number = NUMBER_RE.match
encoding = context.encoding
strict = context.strict
parse_float = context.parse_float
parse_int = context.parse_int
parse_constant = context.parse_constant
object_hook = context.object_hook
def _scan_once(string, idx):
try:
nextchar = string[idx]
except IndexError:
raise StopIteration
if nextchar == '"':
return parse_string(string, idx + 1, encoding, strict)
elif nextchar == '{':
return parse_object((string, idx + 1), encoding, strict, _scan_once, object_hook)
elif nextchar == '[':
return parse_array((string, idx + 1), _scan_once)
elif nextchar == 'n' and string[idx:idx + 4] == 'null':
return None, idx + 4
elif nextchar == 't' and string[idx:idx + 4] == 'true':
return True, idx + 4
elif nextchar == 'f' and string[idx:idx + 5] == 'false':
return False, idx + 5
m = match_number(string, idx)
if m is not None:
integer, frac, exp = m.groups()
if frac or exp:
res = parse_float(integer + (frac or '') + (exp or ''))
else:
res = parse_int(integer)
return res, m.end()
elif nextchar == 'N' and string[idx:idx + 3] == 'NaN':
return parse_constant('NaN'), idx + 3
elif nextchar == 'I' and string[idx:idx + 8] == 'Infinity':
return parse_constant('Infinity'), idx + 8
elif nextchar == '-' and string[idx:idx + 9] == '-Infinity':
return parse_constant('-Infinity'), idx + 9
else:
raise StopIteration
return _scan_once
make_scanner = c_make_scanner or py_make_scanner
mitogen-0.2.9/LICENSE → mitogen-0.3.7/LICENSE
View file @ 985463fd
Copyright 2019, David Wilson
Copyright 2021, the Mitogen authors
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
......
mitogen-0.2.9/mitogen.egg-info/PKG-INFO → mitogen-0.3.7/PKG-INFO
View file @ 985463fd
Metadata-Version: 1.1
Metadata-Version: 2.1
Name: mitogen
Version: 0.2.9
Version: 0.3.7
Summary: Library for writing distributed self-replicating programs.
Home-page: https://github.com/dw/mitogen/
Home-page: https://github.com/mitogen-hq/mitogen/
Author: David Wilson
Author-email: UNKNOWN
License: New BSD
Description: UNKNOWN
Platform: UNKNOWN
Classifier: Environment :: Console
Classifier: Framework :: Ansible
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2.4
Classifier: Programming Language :: Python :: 2.5
Classifier: Programming Language :: Python :: 2.6
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: System :: Distributed Computing
Classifier: Topic :: System :: Systems Administration
Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*
Description-Content-Type: text/markdown
License-File: LICENSE
# Mitogen
<a href="https://mitogen.networkgenomics.com/">Please see the documentation</a>.
![](https://i.imgur.com/eBM6LhJ.gif)
[![Total alerts](https://img.shields.io/lgtm/alerts/g/mitogen-hq/mitogen.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/mitogen-hq/mitogen/alerts/)
[![Build Status](https://dev.azure.com/mitogen-hq/mitogen/_apis/build/status/mitogen-hq.mitogen?branchName=master)](https://dev.azure.com/mitogen-hq/mitogen/_build/latest?definitionId=1&branchName=master)
mitogen-0.3.7/README.md 0 → 100644
View file @ 985463fd
# Mitogen
<a href="https://mitogen.networkgenomics.com/">Please see the documentation</a>.
![](https://i.imgur.com/eBM6LhJ.gif)
[![Total alerts](https://img.shields.io/lgtm/alerts/g/mitogen-hq/mitogen.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/mitogen-hq/mitogen/alerts/)
[![Build Status](https://dev.azure.com/mitogen-hq/mitogen/_apis/build/status/mitogen-hq.mitogen?branchName=master)](https://dev.azure.com/mitogen-hq/mitogen/_build/latest?definitionId=1&branchName=master)
mitogen-0.2.9/ansible_mitogen/affinity.py → mitogen-0.3.7/ansible_mitogen/affinity.py
View file @ 985463fd
......@@ -73,7 +73,9 @@ necessarily involves preventing the scheduler from making load balancing
decisions.
"""
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import ctypes
import logging
import mmap
......
mitogen-0.2.9/ansible_mitogen/connection.py → mitogen-0.3.7/ansible_mitogen/connection.py
View file @ 985463fd
......@@ -26,8 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
from __future__ import unicode_literals
__metaclass__ = type
import errno
import logging
......@@ -40,11 +41,8 @@ import time
import ansible.constants as C
import ansible.errors
import ansible.plugins.connection
import ansible.utils.shlex
import mitogen.core
import mitogen.fork
import mitogen.utils
import ansible_mitogen.mixins
import ansible_mitogen.parsing
......@@ -52,6 +50,7 @@ import ansible_mitogen.process
import ansible_mitogen.services
import ansible_mitogen.target
import ansible_mitogen.transport_config
import ansible_mitogen.utils.unsafe
LOG = logging.getLogger(__name__)
......@@ -183,7 +182,7 @@ def _connect_docker(spec):
'kwargs': {
'username': spec.remote_user(),
'container': spec.remote_addr(),
'python_path': spec.python_path(),
'python_path': spec.python_path(rediscover_python=True),
'connect_timeout': spec.ansible_ssh_timeout() or spec.timeout(),
'remote_name': get_remote_name(spec),
}
......@@ -262,6 +261,21 @@ def _connect_machinectl(spec):
return _connect_setns(spec, kind='machinectl')
def _connect_podman(spec):
"""
Return ContextService arguments for a Docker connection.
"""
return {
'method': 'podman',
'kwargs': {
'username': spec.remote_user(),
'container': spec.remote_addr(),
'python_path': spec.python_path(rediscover_python=True),
'connect_timeout': spec.ansible_ssh_timeout() or spec.timeout(),
'remote_name': get_remote_name(spec),
}
}
def _connect_setns(spec, kind=None):
"""
Return ContextService arguments for a mitogen_setns connection.
......@@ -400,6 +414,7 @@ CONNECTION_METHOD = {
'lxc': _connect_lxc,
'lxd': _connect_lxd,
'machinectl': _connect_machinectl,
'podman': _connect_podman,
'setns': _connect_setns,
'ssh': _connect_ssh,
'smart': _connect_ssh, # issue #548.
......@@ -469,6 +484,7 @@ class Connection(ansible.plugins.connection.ConnectionBase):
login_context = None
#: Only sudo, su, and doas are supported for now.
# Ansible ConnectionBase attribute, removed in Ansible >= 2.8
become_methods = ['sudo', 'su', 'doas']
#: Dict containing init_child() return value as recorded at startup by
......@@ -503,14 +519,8 @@ class Connection(ansible.plugins.connection.ConnectionBase):
#: matching vanilla Ansible behaviour.
loader_basedir = None
def __del__(self):
"""
Ansible cannot be trusted to always call close() e.g. the synchronize
action constructs a local connection like this. So provide a destructor
in the hopes of catching these cases.
"""
# https://github.com/dw/mitogen/issues/140
self.close()
# set by `_get_task_vars()` for interpreter discovery
_action = None
def on_action_run(self, task_vars, delegate_to_hostname, loader_basedir):
"""
......@@ -551,6 +561,23 @@ class Connection(ansible.plugins.connection.ConnectionBase):
connection passed into any running action.
"""
if self._task_vars is not None:
# check for if self._action has already been set or not
# there are some cases where the ansible executor passes in task_vars
# so we don't walk the stack to find them
# TODO: is there a better way to get the ActionModuleMixin object?
# ansible python discovery needs it to run discover_interpreter()
if not isinstance(self._action, ansible_mitogen.mixins.ActionModuleMixin):
f = sys._getframe()
while f:
if f.f_code.co_name == 'run':
f_self = f.f_locals.get('self')
if isinstance(f_self, ansible_mitogen.mixins.ActionModuleMixin):
self._action = f_self
break
elif f.f_code.co_name == '_execute_meta':
break
f = f.f_back
return self._task_vars
f = sys._getframe()
......@@ -559,6 +586,9 @@ class Connection(ansible.plugins.connection.ConnectionBase):
f_locals = f.f_locals
f_self = f_locals.get('self')
if isinstance(f_self, ansible_mitogen.mixins.ActionModuleMixin):
# backref for python interpreter discovery, should be safe because _get_task_vars
# is always called before running interpreter discovery
self._action = f_self
task_vars = f_locals.get('task_vars')
if task_vars:
LOG.debug('recovered task_vars from Action')
......@@ -600,16 +630,33 @@ class Connection(ansible.plugins.connection.ConnectionBase):
does not make sense to extract connection-related configuration for the
delegated-to machine from them.
"""
def _fetch_task_var(task_vars, key):
"""
Special helper func in case vars can be templated
"""
SPECIAL_TASK_VARS = [
'ansible_python_interpreter'
]
if key in task_vars:
val = task_vars[key]
if '{' in str(val) and key in SPECIAL_TASK_VARS:
# template every time rather than storing in a cache
# in case a different template value is used in a different task
val = self.templar.template(
val,
preserve_trailing_newlines=True,
escape_backslashes=False
)
return val
task_vars = self._get_task_vars()
if self.delegate_to_hostname is None:
if key in task_vars:
return task_vars[key]
return _fetch_task_var(task_vars, key)
else:
delegated_vars = task_vars['ansible_delegated_vars']
if self.delegate_to_hostname in delegated_vars:
task_vars = delegated_vars[self.delegate_to_hostname]
if key in task_vars:
return task_vars[key]
return _fetch_task_var(task_vars, key)
return default
......@@ -629,6 +676,9 @@ class Connection(ansible.plugins.connection.ConnectionBase):
@property
def connected(self):
"""
Ansible connection plugin property. Used by ansible-connection command.
"""
return self.context is not None
def _spec_from_via(self, proxied_inventory_name, via_spec):
......@@ -654,6 +704,8 @@ class Connection(ansible.plugins.connection.ConnectionBase):
inventory_name=inventory_name,
play_context=self._play_context,
host_vars=dict(via_vars), # TODO: make it lazy
task_vars=self._get_task_vars(), # needed for interpreter discovery in parse_python_path
action=self._action,
become_method=become_method or None,
become_user=become_user or None,
)
......@@ -745,7 +797,7 @@ class Connection(ansible.plugins.connection.ConnectionBase):
call_context=self.binding.get_service_context(),
service_name='ansible_mitogen.services.ContextService',
method_name='get',
stack=mitogen.utils.cast(list(stack)),
stack=ansible_mitogen.utils.unsafe.cast(list(stack)),
)
except mitogen.core.CallError:
LOG.warning('Connection failed; stack configuration was:\n%s',
......@@ -785,14 +837,18 @@ class Connection(ansible.plugins.connection.ConnectionBase):
the _connect_*() service calls defined above to cause the master
process to establish the real connection on our behalf, or return a
reference to the existing one.
Ansible connection plugin method.
"""
# In some Ansible connection plugins this method returns self.
# However nothing I've found uses it, it's not even assigned.
if self.connected:
return
inventory_name, stack = self._build_stack()
worker_model = ansible_mitogen.process.get_worker_model()
self.binding = worker_model.get_binding(
mitogen.utils.cast(inventory_name)
ansible_mitogen.utils.unsafe.cast(inventory_name)
)
self._connect_stack(stack)
......@@ -823,6 +879,8 @@ class Connection(ansible.plugins.connection.ConnectionBase):
Arrange for the mitogen.master.Router running in the worker to
gracefully shut down, and wait for shutdown to complete. Safe to call
multiple times.
Ansible connection plugin method.
"""
self._put_connection()
if self.binding:
......@@ -839,6 +897,8 @@ class Connection(ansible.plugins.connection.ConnectionBase):
any local state we hold for the connection, returns the Connection to
the 'disconnected' state, and informs ContextService the connection is
bad somehow, and should be shut down and discarded.
Ansible connection plugin method.
"""
if self._play_context.remote_addr is None:
# <2.5.6 incorrectly populate PlayContext for reset_connection
......@@ -847,6 +907,18 @@ class Connection(ansible.plugins.connection.ConnectionBase):
self.reset_compat_msg
)
# Strategy's _execute_meta doesn't have an action obj but we'll need one for
# running interpreter_discovery
# will create a new temporary action obj for this purpose
self._action = ansible_mitogen.mixins.ActionModuleMixin(
task=0,
connection=self,
play_context=self._play_context,
loader=0,
templar=0,
shared_loader_obj=0
)
# Clear out state in case we were ever connected.
self.close()
......@@ -861,7 +933,7 @@ class Connection(ansible.plugins.connection.ConnectionBase):
call_context=binding.get_service_context(),
service_name='ansible_mitogen.services.ContextService',
method_name='reset',
stack=mitogen.utils.cast(list(stack)),
stack=ansible_mitogen.utils.unsafe.cast(list(stack)),
)
finally:
binding.close()
......@@ -933,12 +1005,14 @@ class Connection(ansible.plugins.connection.ConnectionBase):
Data to supply on ``stdin`` of the process.
:returns:
(return code, stdout bytes, stderr bytes)
Ansible connection plugin method.
"""
emulate_tty = (not in_data and sudoable)
rc, stdout, stderr = self.get_chain().call(
ansible_mitogen.target.exec_command,
cmd=mitogen.utils.cast(cmd),
in_data=mitogen.utils.cast(in_data),
cmd=ansible_mitogen.utils.unsafe.cast(cmd),
in_data=ansible_mitogen.utils.unsafe.cast(in_data),
chdir=mitogen_chdir or self.get_default_cwd(),
emulate_tty=emulate_tty,
)
......@@ -958,12 +1032,14 @@ class Connection(ansible.plugins.connection.ConnectionBase):
Remote filesystem path to read.
:param str out_path:
Local filesystem path to write.
Ansible connection plugin method.
"""
self._connect()
ansible_mitogen.target.transfer_file(
context=self.context,
# in_path may be AnsibleUnicode
in_path=mitogen.utils.cast(in_path),
in_path=ansible_mitogen.utils.unsafe.cast(in_path),
out_path=out_path
)
......@@ -981,7 +1057,7 @@ class Connection(ansible.plugins.connection.ConnectionBase):
"""
self.get_chain().call_no_reply(
ansible_mitogen.target.write_path,
mitogen.utils.cast(out_path),
ansible_mitogen.utils.unsafe.cast(out_path),
mitogen.core.Blob(data),
mode=mode,
utimes=utimes,
......@@ -1007,6 +1083,8 @@ class Connection(ansible.plugins.connection.ConnectionBase):
Local filesystem path to read.
:param str out_path:
Remote filesystem path to write.
Ansible connection plugin method.
"""
try:
st = os.stat(in_path)
......@@ -1027,7 +1105,7 @@ class Connection(ansible.plugins.connection.ConnectionBase):
s = fp.read(self.SMALL_FILE_LIMIT + 1)
finally:
fp.close()
except OSError:
except OSError as e:
self._throw_io_error(e, in_path)
raise
......@@ -1041,7 +1119,7 @@ class Connection(ansible.plugins.connection.ConnectionBase):
call_context=self.binding.get_service_context(),
service_name='mitogen.service.FileService',
method_name='register',
path=mitogen.utils.cast(in_path)
path=ansible_mitogen.utils.unsafe.cast(in_path)
)
# For now this must remain synchronous, as the action plug-in may have
......
mitogen-0.2.9/ansible_mitogen/loaders.py → mitogen-0.3.7/ansible_mitogen/loaders.py
View file @ 985463fd
......@@ -30,10 +30,16 @@
Stable names for PluginLoader instances across Ansible versions.
"""
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import ansible.errors
import ansible_mitogen.utils
__all__ = [
'action_loader',
'become_loader',
'connection_loader',
'module_loader',
'module_utils_loader',
......@@ -41,22 +47,57 @@ __all__ = [
'strategy_loader',
]
try:
from ansible.plugins.loader import action_loader
from ansible.plugins.loader import connection_loader
from ansible.plugins.loader import module_loader
from ansible.plugins.loader import module_utils_loader
from ansible.plugins.loader import shell_loader
from ansible.plugins.loader import strategy_loader
except ImportError: # Ansible <2.4
from ansible.plugins import action_loader
from ansible.plugins import connection_loader
from ansible.plugins import module_loader
from ansible.plugins import module_utils_loader
from ansible.plugins import shell_loader
from ansible.plugins import strategy_loader
ANSIBLE_VERSION_MIN = (2, 10)
ANSIBLE_VERSION_MAX = (2, 16)
NEW_VERSION_MSG = (
"Your Ansible version (%s) is too recent. The most recent version\n"
"supported by Mitogen for Ansible is %s.x. Please check the Mitogen\n"
"release notes to see if a new version is available, otherwise\n"
"subscribe to the corresponding GitHub issue to be notified when\n"
"support becomes available.\n"
"\n"
" https://mitogen.rtfd.io/en/latest/changelog.html\n"
" https://github.com/mitogen-hq/mitogen/issues/\n"
)
OLD_VERSION_MSG = (
"Your version of Ansible (%s) is too old. The oldest version supported by "
"Mitogen for Ansible is %s."
)
def assert_supported_release():
"""
Throw AnsibleError with a descriptive message in case of being loaded into
an unsupported Ansible release.
"""
v = ansible_mitogen.utils.ansible_version
if v[:2] < ANSIBLE_VERSION_MIN:
raise ansible.errors.AnsibleError(
OLD_VERSION_MSG % (v, ANSIBLE_VERSION_MIN)
)
if v[:2] > ANSIBLE_VERSION_MAX:
raise ansible.errors.AnsibleError(
NEW_VERSION_MSG % (v, ANSIBLE_VERSION_MAX)
)
# this is the first file our strategy plugins import, so we need to check this here
# in prior Ansible versions, connection_loader.get_with_context didn't exist, so if a user
# is trying to load an old Ansible version, we'll fail and error gracefully
assert_supported_release()
from ansible.plugins.loader import action_loader
from ansible.plugins.loader import become_loader
from ansible.plugins.loader import connection_loader
from ansible.plugins.loader import module_loader
from ansible.plugins.loader import module_utils_loader
from ansible.plugins.loader import shell_loader
from ansible.plugins.loader import strategy_loader
# These are original, unwrapped implementations
action_loader__get = action_loader.get
connection_loader__get = connection_loader.get
connection_loader__get = connection_loader.get_with_context
mitogen-0.2.9/ansible_mitogen/logging.py → mitogen-0.3.7/ansible_mitogen/logging.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import logging
import os
......@@ -36,8 +38,8 @@ import mitogen.utils
try:
from __main__ import display
except ImportError:
from ansible.utils.display import Display
display = Display()
import ansible.utils.display
display = ansible.utils.display.Display()
#: The process name set via :func:`set_process_name`.
......
mitogen-0.2.9/ansible_mitogen/mixins.py → mitogen-0.3.7/ansible_mitogen/mixins.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import logging
import os
import pwd
......@@ -48,11 +50,13 @@ import ansible.plugins.action
import mitogen.core
import mitogen.select
import mitogen.utils
import ansible_mitogen.connection
import ansible_mitogen.planner
import ansible_mitogen.target
import ansible_mitogen.utils
import ansible_mitogen.utils.unsafe
from ansible.module_utils._text import to_text
try:
......@@ -60,6 +64,17 @@ try:
except ImportError:
from ansible.vars.unsafe_proxy import wrap_var
try:
# ansible 2.8 moved remove_internal_keys to the clean module
from ansible.vars.clean import remove_internal_keys
except ImportError:
try:
from ansible.vars.manager import remove_internal_keys
except ImportError:
# ansible 2.3.3 has remove_internal_keys as a protected func on the action class
# we'll fallback to calling self._remove_internal_keys in this case
remove_internal_keys = lambda a: "Not found"
LOG = logging.getLogger(__name__)
......@@ -108,6 +123,16 @@ class ActionModuleMixin(ansible.plugins.action.ActionBase):
if not isinstance(connection, ansible_mitogen.connection.Connection):
_, self.__class__ = type(self).__bases__
# required for python interpreter discovery
connection.templar = self._templar
self._finding_python_interpreter = False
self._rediscovered_python = False
# redeclaring interpreter discovery vars here in case running ansible < 2.8.0
self._discovered_interpreter_key = None
self._discovered_interpreter = False
self._discovery_deprecation_warnings = []
self._discovery_warnings = []
def run(self, tmp=None, task_vars=None):
"""
Override run() to notify Connection of task-specific data, so it has a
......@@ -162,7 +187,7 @@ class ActionModuleMixin(ansible.plugins.action.ActionBase):
LOG.debug('_remote_file_exists(%r)', path)
return self._connection.get_chain().call(
ansible_mitogen.target.file_exists,
mitogen.utils.cast(path)
ansible_mitogen.utils.unsafe.cast(path)
)
def _configure_module(self, module_name, module_args, task_vars=None):
......@@ -205,7 +230,7 @@ class ActionModuleMixin(ansible.plugins.action.ActionBase):
with a pipelined call to :func:`ansible_mitogen.target.prune_tree`.
"""
LOG.debug('_remove_tmp_path(%r)', tmp_path)
if tmp_path is None and ansible.__version__ > '2.6':
if tmp_path is None and ansible_mitogen.utils.ansible_version[:2] >= (2, 6):
tmp_path = self._connection._shell.tmpdir # 06f73ad578d
if tmp_path is not None:
self._connection.get_chain().call_no_reply(
......@@ -299,7 +324,7 @@ class ActionModuleMixin(ansible.plugins.action.ActionBase):
# ~root/.ansible -> /root/.ansible
return self._connection.get_chain(use_login=(not sudoable)).call(
os.path.expanduser,
mitogen.utils.cast(path),
ansible_mitogen.utils.unsafe.cast(path),
)
def get_task_timeout_secs(self):
......@@ -314,7 +339,7 @@ class ActionModuleMixin(ansible.plugins.action.ActionBase):
def _set_temp_file_args(self, module_args, wrap_async):
# Ansible>2.5 module_utils reuses the action's temporary directory if
# one exists. Older versions error if this key is present.
if ansible.__version__ > '2.5':
if ansible_mitogen.utils.ansible_version[:2] >= (2, 5):
if wrap_async:
# Sharing is not possible with async tasks, as in that case,
# the directory must outlive the action plug-in.
......@@ -325,7 +350,7 @@ class ActionModuleMixin(ansible.plugins.action.ActionBase):
# If _ansible_tmpdir is unset, Ansible>2.6 module_utils will use
# _ansible_remote_tmp as the location to create the module's temporary
# directory. Older versions error if this key is present.
if ansible.__version__ > '2.6':
if ansible_mitogen.utils.ansible_version[:2] >= (2, 6):
module_args['_ansible_remote_tmp'] = (
self._connection.get_good_temp_dir()
)
......@@ -350,26 +375,61 @@ class ActionModuleMixin(ansible.plugins.action.ActionBase):
self._compute_environment_string(env)
self._set_temp_file_args(module_args, wrap_async)
# there's a case where if a task shuts down the node and then immediately calls
# wait_for_connection, the `ping` test from Ansible won't pass because we lost connection
# clearing out context forces a reconnect
# see https://github.com/dw/mitogen/issues/655 and Ansible's `wait_for_connection` module for more info
if module_name == 'ansible.legacy.ping' and type(self).__name__ == 'wait_for_connection':
self._connection.context = None
self._connection._connect()
result = ansible_mitogen.planner.invoke(
ansible_mitogen.planner.Invocation(
action=self,
connection=self._connection,
module_name=mitogen.core.to_text(module_name),
module_args=mitogen.utils.cast(module_args),
module_name=ansible_mitogen.utils.unsafe.cast(mitogen.core.to_text(module_name)),
module_args=ansible_mitogen.utils.unsafe.cast(module_args),
task_vars=task_vars,
templar=self._templar,
env=mitogen.utils.cast(env),
env=ansible_mitogen.utils.unsafe.cast(env),
wrap_async=wrap_async,
timeout_secs=self.get_task_timeout_secs(),
)
)
if tmp and ansible.__version__ < '2.5' and delete_remote_tmp:
if tmp and delete_remote_tmp and ansible_mitogen.utils.ansible_version[:2] < (2, 5):
# Built-in actions expected tmpdir to be cleaned up automatically
# on _execute_module().
self._remove_tmp_path(tmp)
# prevents things like discovered_interpreter_* or ansible_discovered_interpreter_* from being set
# handle ansible 2.3.3 that has remove_internal_keys in a different place
check = remove_internal_keys(result)
if check == 'Not found':
self._remove_internal_keys(result)
# taken from _execute_module of ansible 2.8.6
# propagate interpreter discovery results back to the controller
if self._discovered_interpreter_key:
if result.get('ansible_facts') is None:
result['ansible_facts'] = {}
# only cache discovered_interpreter if we're not running a rediscovery
# rediscovery happens in places like docker connections that could have different
# python interpreters than the main host
if not self._rediscovered_python:
result['ansible_facts'][self._discovered_interpreter_key] = self._discovered_interpreter
if self._discovery_warnings:
if result.get('warnings') is None:
result['warnings'] = []
result['warnings'].extend(self._discovery_warnings)
if self._discovery_deprecation_warnings:
if result.get('deprecations') is None:
result['deprecations'] = []
result['deprecations'].extend(self._discovery_deprecation_warnings)
return wrap_var(result)
def _postprocess_response(self, result):
......@@ -407,17 +467,54 @@ class ActionModuleMixin(ansible.plugins.action.ActionBase):
"""
LOG.debug('_low_level_execute_command(%r, in_data=%r, exe=%r, dir=%r)',
cmd, type(in_data), executable, chdir)
if executable is None: # executable defaults to False
executable = self._play_context.executable
if executable:
cmd = executable + ' -c ' + shlex_quote(cmd)
rc, stdout, stderr = self._connection.exec_command(
cmd=cmd,
in_data=in_data,
sudoable=sudoable,
mitogen_chdir=chdir,
)
# TODO: HACK: if finding python interpreter then we need to keep
# calling exec_command until we run into the right python we'll use
# chicken-and-egg issue, mitogen needs a python to run low_level_execute_command
# which is required by Ansible's discover_interpreter function
if self._finding_python_interpreter:
possible_pythons = [
'/usr/bin/python',
'python3',
'python3.7',
'python3.6',
'python3.5',
'python2.7',
'python2.6',
'/usr/libexec/platform-python',
'/usr/bin/python3',
'python'
]
else:
# not used, just adding a filler value
possible_pythons = ['python']
def _run_cmd():
return self._connection.exec_command(
cmd=cmd,
in_data=in_data,
sudoable=sudoable,
mitogen_chdir=chdir,
)
for possible_python in possible_pythons:
try:
self._possible_python_interpreter = possible_python
rc, stdout, stderr = _run_cmd()
# TODO: what exception is thrown?
except:
# we've reached the last python attempted and failed
# TODO: could use enumerate(), need to check which version of python first had it though
if possible_python == 'python':
raise
else:
continue
stdout_text = to_text(stdout, errors=encoding_errors)
return {
......
mitogen-0.2.9/ansible_mitogen/module_finder.py → mitogen-0.3.7/ansible_mitogen/module_finder.py
View file @ 985463fd
......@@ -26,19 +26,36 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
from __future__ import unicode_literals
__metaclass__ = type
import collections
import imp
import logging
import os
import re
import sys
try:
# Python >= 3.4, PEP 451 ModuleSpec API
import importlib.machinery
import importlib.util
except ImportError:
# Python < 3.4, PEP 302 Import Hooks
import imp
import mitogen.master
LOG = logging.getLogger(__name__)
PREFIX = 'ansible.module_utils.'
# Analog of `importlib.machinery.ModuleSpec` or `pkgutil.ModuleInfo`.
# name Unqualified name of the module.
# path Filesystem path of the module.
# kind One of the constants in `imp`, as returned in `imp.find_module()`
# parent `ansible_mitogen.module_finder.Module` of parent package (if any).
Module = collections.namedtuple('Module', 'name path kind parent')
......@@ -118,14 +135,121 @@ def find_relative(parent, name, path=()):
def scan_fromlist(code):
"""Return an iterator of (level, name) for explicit imports in a code
object.
Not all names identify a module. `from os import name, path` generates
`(0, 'os.name'), (0, 'os.path')`, but `os.name` is usually a string.
>>> src = 'import a; import b.c; from d.e import f; from g import h, i\\n'
>>> code = compile(src, '<str>', 'exec')
>>> list(scan_fromlist(code))
[(0, 'a'), (0, 'b.c'), (0, 'd.e.f'), (0, 'g.h'), (0, 'g.i')]
"""
for level, modname_s, fromlist in mitogen.master.scan_code_imports(code):
for name in fromlist:
yield level, '%s.%s' % (modname_s, name)
yield level, str('%s.%s' % (modname_s, name))
if not fromlist:
yield level, modname_s
def walk_imports(code, prefix=None):
"""Return an iterator of names for implicit parent imports & explicit
imports in a code object.
If a prefix is provided, then only children of that prefix are included.
Not all names identify a module. `from os import name, path` generates
`'os', 'os.name', 'os.path'`, but `os.name` is usually a string.
>>> source = 'import a; import b; import b.c; from b.d import e, f\\n'
>>> code = compile(source, '<str>', 'exec')
>>> list(walk_imports(code))
['a', 'b', 'b', 'b.c', 'b', 'b.d', 'b.d.e', 'b.d.f']
>>> list(walk_imports(code, prefix='b'))
['b.c', 'b.d', 'b.d.e', 'b.d.f']
"""
if prefix is None:
prefix = ''
pattern = re.compile(r'(^|\.)(\w+)')
start = len(prefix)
for _, name, fromlist in mitogen.master.scan_code_imports(code):
if not name.startswith(prefix):
continue
for match in pattern.finditer(name, start):
yield name[:match.end()]
for leaf in fromlist:
yield str('%s.%s' % (name, leaf))
def scan(module_name, module_path, search_path):
# type: (str, str, list[str]) -> list[(str, str, bool)]
"""Return a list of (name, path, is_package) for ansible.module_utils
imports used by an Ansible module.
"""
log = LOG.getChild('scan')
log.debug('%r, %r, %r', module_name, module_path, search_path)
if sys.version_info >= (3, 4):
result = _scan_importlib_find_spec(
module_name, module_path, search_path,
)
log.debug('_scan_importlib_find_spec %r', result)
else:
result = _scan_imp_find_module(module_name, module_path, search_path)
log.debug('_scan_imp_find_module %r', result)
return result
def _scan_importlib_find_spec(module_name, module_path, search_path):
# type: (str, str, list[str]) -> list[(str, str, bool)]
module = importlib.machinery.ModuleSpec(
module_name, loader=None, origin=module_path,
)
prefix = importlib.machinery.ModuleSpec(
PREFIX.rstrip('.'), loader=None,
)
prefix.submodule_search_locations = search_path
queue = collections.deque([module])
specs = {prefix.name: prefix}
while queue:
spec = queue.popleft()
if spec.origin is None:
continue
try:
with open(spec.origin, 'rb') as f:
code = compile(f.read(), spec.name, 'exec')
except Exception as exc:
raise ValueError((exc, module, spec, specs))
for name in walk_imports(code, prefix.name):
if name in specs:
continue
parent_name = name.rpartition('.')[0]
parent = specs[parent_name]
if parent is None or not parent.submodule_search_locations:
specs[name] = None
continue
child = importlib.util._find_spec(
name, parent.submodule_search_locations,
)
if child is None or child.origin is None:
specs[name] = None
continue
specs[name] = child
queue.append(child)
del specs[prefix.name]
return sorted(
(spec.name, spec.origin, spec.submodule_search_locations is not None)
for spec in specs.values() if spec is not None
)
def _scan_imp_find_module(module_name, module_path, search_path):
# type: (str, str, list[str]) -> list[(str, str, bool)]
module = Module(module_name, module_path, imp.PY_SOURCE, None)
stack = [module]
seen = set()
......
mitogen-0.2.9/ansible_mitogen/parsing.py → mitogen-0.3.7/ansible_mitogen/parsing.py
View file @ 985463fd
......@@ -26,8 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
from __future__ import unicode_literals
__metaclass__ = type
import mitogen.core
......
mitogen-0.2.9/ansible_mitogen/planner.py → mitogen-0.3.7/ansible_mitogen/planner.py
View file @ 985463fd
......@@ -34,18 +34,20 @@ files/modules known missing.
[0] "Ansible Module Architecture", developing_program_flow_modules.html
"""
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
from __future__ import unicode_literals
__metaclass__ = type
import json
import logging
import os
import random
import re
from ansible.executor import module_common
import ansible.collections.list
import ansible.errors
import ansible.module_utils
import ansible.release
import ansible.executor.module_common
import mitogen.core
import mitogen.select
......@@ -57,7 +59,8 @@ import ansible_mitogen.target
LOG = logging.getLogger(__name__)
NO_METHOD_MSG = 'Mitogen: no invocation method found for: '
NO_INTERPRETER_MSG = 'module (%s) is missing interpreter line'
NO_MODULE_MSG = 'The module %s was not found in configured module paths.'
# NOTE: Ansible 2.10 no longer has a `.` at the end of NO_MODULE_MSG error
NO_MODULE_MSG = 'The module %s was not found in configured module paths'
_planner_by_path = {}
......@@ -96,6 +99,13 @@ class Invocation(object):
#: Initially ``None``, but set by :func:`invoke`. The raw source or
#: binary contents of the module.
self._module_source = None
#: Initially ``{}``, but set by :func:`invoke`. Optional source to send
#: to :func:`propagate_paths_and_modules` to fix Python3.5 relative import errors
self._overridden_sources = {}
#: Initially ``set()``, but set by :func:`invoke`. Optional source paths to send
#: to :func:`propagate_paths_and_modules` to handle loading source dependencies from
#: places outside of the main source path, such as collections
self._extra_sys_paths = set()
def get_module_source(self):
if self._module_source is None:
......@@ -182,7 +192,7 @@ class BinaryPlanner(Planner):
@classmethod
def detect(cls, path, source):
return module_common._is_binary(source)
return ansible.executor.module_common._is_binary(source)
def get_push_files(self):
return [mitogen.core.to_text(self._inv.module_path)]
......@@ -259,7 +269,7 @@ class JsonArgsPlanner(ScriptPlanner):
@classmethod
def detect(cls, path, source):
return module_common.REPLACER_JSONARGS in source
return ansible.executor.module_common.REPLACER_JSONARGS in source
class WantJsonPlanner(ScriptPlanner):
......@@ -288,11 +298,11 @@ class NewStylePlanner(ScriptPlanner):
preprocessing the module.
"""
runner_name = 'NewStyleRunner'
marker = b'from ansible.module_utils.'
MARKER = re.compile(br'from ansible(?:_collections|\.module_utils)\.')
@classmethod
def detect(cls, path, source):
return cls.marker in source
return cls.MARKER.search(source) is not None
def _get_interpreter(self):
return None, None
......@@ -312,6 +322,8 @@ class NewStylePlanner(ScriptPlanner):
ALWAYS_FORK_MODULES = frozenset([
'dnf', # issue #280; py-dnf/hawkey need therapy
'firewalld', # issue #570: ansible module_utils caches dbus conn
'ansible.legacy.dnf', # issue #776
'ansible.builtin.dnf', # issue #832
])
def should_fork(self):
......@@ -351,7 +363,7 @@ class NewStylePlanner(ScriptPlanner):
module_name='ansible_module_%s' % (self._inv.module_name,),
module_path=self._inv.module_path,
search_path=self.get_search_path(),
builtin_path=module_common._MODULE_UTILS_PATH,
builtin_path=ansible.executor.module_common._MODULE_UTILS_PATH,
context=self._inv.connection.context,
)
return self._module_map
......@@ -394,7 +406,7 @@ class ReplacerPlanner(NewStylePlanner):
@classmethod
def detect(cls, path, source):
return module_common.REPLACER in source
return ansible.executor.module_common.REPLACER in source
class OldStylePlanner(ScriptPlanner):
......@@ -416,36 +428,21 @@ _planners = [
]
try:
_get_ansible_module_fqn = module_common._get_ansible_module_fqn
except AttributeError:
_get_ansible_module_fqn = None
def py_modname_from_path(name, path):
"""
Fetch the logical name of a new-style module as it might appear in
:data:`sys.modules` of the target's Python interpreter.
* For Ansible <2.7, this is an unpackaged module named like
"ansible_module_%s".
* For Ansible <2.9, this is an unpackaged module named like
"ansible.modules.%s"
* Since Ansible 2.9, modules appearing within a package have the original
package hierarchy approximated on the target, enabling relative imports
to function correctly. For example, "ansible.modules.system.setup".
"""
# 2.9+
if _get_ansible_module_fqn:
try:
return _get_ansible_module_fqn(path)
except ValueError:
pass
if ansible.__version__ < '2.7':
return 'ansible_module_' + name
try:
return ansible.executor.module_common._get_ansible_module_fqn(path)
except AttributeError:
pass
except ValueError:
pass
return 'ansible.modules.' + name
......@@ -475,7 +472,10 @@ def _propagate_deps(invocation, planner, context):
context=context,
paths=planner.get_push_files(),
modules=planner.get_module_deps(),
# modules=planner.get_module_deps(), TODO
overridden_sources=invocation._overridden_sources,
# needs to be a list because can't unpickle() a set()
extra_sys_paths=list(invocation._extra_sys_paths),
)
......@@ -524,18 +524,52 @@ def _invoke_isolated_task(invocation, planner):
context.shutdown()
def _get_planner(name, path, source):
def _get_planner(invocation, source):
for klass in _planners:
if klass.detect(path, source):
LOG.debug('%r accepted %r (filename %r)', klass, name, path)
if klass.detect(invocation.module_path, source):
LOG.debug(
'%r accepted %r (filename %r)',
klass, invocation.module_name, invocation.module_path,
)
return klass
LOG.debug('%r rejected %r', klass, name)
LOG.debug('%r rejected %r', klass, invocation.module_name)
raise ansible.errors.AnsibleError(NO_METHOD_MSG + repr(invocation))
def _fix_py35(invocation, module_source):
"""
super edge case with a relative import error in Python 3.5.1-3.5.3
in Ansible's setup module when using Mitogen
https://github.com/dw/mitogen/issues/672#issuecomment-636408833
We replace a relative import in the setup module with the actual full file path
This works in vanilla Ansible but not in Mitogen otherwise
"""
if invocation.module_name in {'ansible.builtin.setup', 'ansible.legacy.setup', 'setup'} and \
invocation.module_path not in invocation._overridden_sources:
# in-memory replacement of setup module's relative import
# would check for just python3.5 and run this then but we don't know the
# target python at this time yet
# NOTE: another ansible 2.10-specific fix: `from ..module_utils` used to be `from ...module_utils`
module_source = module_source.replace(
b"from ..module_utils.basic import AnsibleModule",
b"from ansible.module_utils.basic import AnsibleModule"
)
invocation._overridden_sources[invocation.module_path] = module_source
def _load_collections(invocation):
"""
Special loader that ensures that `ansible_collections` exist as a module path for import
Goes through all collection path possibilities and stores paths to installed collections
Stores them on the current invocation to later be passed to the master service
"""
for collection_path in ansible.collections.list.list_collection_dirs():
invocation._extra_sys_paths.add(collection_path.decode('utf-8'))
def invoke(invocation):
"""
Find a Planner subclass corresnding to `invocation` and use it to invoke
Find a Planner subclass corresponding to `invocation` and use it to invoke
the module.
:param Invocation invocation:
......@@ -555,10 +589,14 @@ def invoke(invocation):
invocation.module_path = mitogen.core.to_text(path)
if invocation.module_path not in _planner_by_path:
if 'ansible_collections' in invocation.module_path:
_load_collections(invocation)
module_source = invocation.get_module_source()
_fix_py35(invocation, module_source)
_planner_by_path[invocation.module_path] = _get_planner(
invocation.module_name,
invocation.module_path,
invocation.get_module_source()
invocation,
module_source
)
planner = _planner_by_path[invocation.module_path](invocation)
......
mitogen-0.2.9/ansible_mitogen/plugins/action/mitogen_fetch.py → mitogen-0.3.7/ansible_mitogen/plugins/action/mitogen_fetch.py
View file @ 985463fd
......@@ -18,23 +18,17 @@ from __future__ import (absolute_import, division, print_function)
__metaclass__ = type
import os
from ansible.module_utils._text import to_bytes
import base64
from ansible.errors import AnsibleError, AnsibleActionFail, AnsibleActionSkip
from ansible.module_utils.common.text.converters import to_bytes, to_text
from ansible.module_utils.six import string_types
from ansible.module_utils.parsing.convert_bool import boolean
from ansible.plugins.action import ActionBase
from ansible.utils.hashing import checksum, md5, secure_hash
from ansible.utils.path import makedirs_safe
from ansible.utils.display import Display
from ansible.utils.hashing import checksum, checksum_s, md5, secure_hash
from ansible.utils.path import makedirs_safe, is_subpath
REMOTE_CHECKSUM_ERRORS = {
'0': "unable to calculate the checksum of the remote file",
'1': "the remote file does not exist",
'2': "no read permission on remote file",
'3': "remote file is a directory, fetch cannot work on directories",
'4': "python isn't present on the system. Unable to compute checksum",
'5': "stdlib json was not found on the remote machine. Only the raw module can work without those installed",
}
display = Display()
class ActionModule(ActionBase):
......@@ -45,36 +39,94 @@ class ActionModule(ActionBase):
task_vars = dict()
result = super(ActionModule, self).run(tmp, task_vars)
del tmp # tmp no longer has any effect
try:
if self._play_context.check_mode:
result['skipped'] = True
result['msg'] = 'check mode not (yet) supported for this module'
return result
raise AnsibleActionSkip('check mode not (yet) supported for this module')
source = self._task.args.get('src', None)
original_dest = dest = self._task.args.get('dest', None)
flat = boolean(self._task.args.get('flat'), strict=False)
fail_on_missing = boolean(self._task.args.get('fail_on_missing', True), strict=False)
validate_checksum = boolean(self._task.args.get('validate_checksum', True), strict=False)
msg = ''
# validate source and dest are strings FIXME: use basic.py and module specs
source = self._task.args.get('src')
if not isinstance(source, string_types):
result['msg'] = "Invalid type supplied for source option, it must be a string"
msg = "Invalid type supplied for source option, it must be a string"
dest = self._task.args.get('dest')
if not isinstance(dest, string_types):
result['msg'] = "Invalid type supplied for dest option, it must be a string"
msg = "Invalid type supplied for dest option, it must be a string"
if source is None or dest is None:
msg = "src and dest are required"
if result.get('msg'):
result['failed'] = True
return result
if msg:
raise AnsibleActionFail(msg)
source = self._connection._shell.join_path(source)
source = self._remote_expand_user(source)
# calculate checksum for the remote file, don't bother if using
# become as slurp will be used Force remote_checksum to follow
# symlinks because fetch always follows symlinks
remote_checksum = self._remote_checksum(source, all_vars=task_vars, follow=True)
remote_stat = {}
remote_checksum = None
if True:
# Get checksum for the remote file even using become. Mitogen doesn't need slurp.
# Follow symlinks because fetch always follows symlinks
try:
remote_stat = self._execute_remote_stat(source, all_vars=task_vars, follow=True)
except AnsibleError as ae:
result['changed'] = False
result['file'] = source
if fail_on_missing:
result['failed'] = True
result['msg'] = to_text(ae)
else:
result['msg'] = "%s, ignored" % to_text(ae, errors='surrogate_or_replace')
return result
remote_checksum = remote_stat.get('checksum')
if remote_stat.get('exists'):
if remote_stat.get('isdir'):
result['failed'] = True
result['changed'] = False
result['msg'] = "remote file is a directory, fetch cannot work on directories"
# Historically, these don't fail because you may want to transfer
# a log file that possibly MAY exist but keep going to fetch other
# log files. Today, this is better achieved by adding
# ignore_errors or failed_when to the task. Control the behaviour
# via fail_when_missing
if not fail_on_missing:
result['msg'] += ", not transferring, ignored"
del result['changed']
del result['failed']
return result
# use slurp if permissions are lacking or privilege escalation is needed
remote_data = None
if remote_checksum in (None, '1', ''):
slurpres = self._execute_module(module_name='ansible.legacy.slurp', module_args=dict(src=source), task_vars=task_vars)
if slurpres.get('failed'):
if not fail_on_missing:
result['file'] = source
result['changed'] = False
else:
result.update(slurpres)
if 'not found' in slurpres.get('msg', ''):
result['msg'] = "the remote file does not exist, not transferring, ignored"
elif slurpres.get('msg', '').startswith('source is a directory'):
result['msg'] = "remote file is a directory, fetch cannot work on directories"
return result
else:
if slurpres['encoding'] == 'base64':
remote_data = base64.b64decode(slurpres['content'])
if remote_data is not None:
remote_checksum = checksum_s(remote_data)
# calculate the destination name
if os.path.sep not in self._connection._shell.join_path('a', ''):
......@@ -83,13 +135,14 @@ class ActionModule(ActionBase):
else:
source_local = source
dest = os.path.expanduser(dest)
# ensure we only use file name, avoid relative paths
if not is_subpath(dest, original_dest):
# TODO: ? dest = os.path.expanduser(dest.replace(('../','')))
raise AnsibleActionFail("Detected directory traversal, expected to be contained in '%s' but got '%s'" % (original_dest, dest))
if flat:
if os.path.isdir(to_bytes(dest, errors='surrogate_or_strict')) and not dest.endswith(os.sep):
result['msg'] = "dest is an existing directory, use a trailing slash if you want to fetch src into that directory"
result['file'] = dest
result['failed'] = True
return result
raise AnsibleActionFail("dest is an existing directory, use a trailing slash if you want to fetch src into that directory")
if dest.endswith(os.sep):
# if the path ends with "/", we'll use the source filename as the
# destination filename
......@@ -106,23 +159,7 @@ class ActionModule(ActionBase):
target_name = self._play_context.remote_addr
dest = "%s/%s/%s" % (self._loader.path_dwim(dest), target_name, source_local)
dest = dest.replace("//", "/")
if remote_checksum in REMOTE_CHECKSUM_ERRORS:
result['changed'] = False
result['file'] = source
result['msg'] = REMOTE_CHECKSUM_ERRORS[remote_checksum]
# Historically, these don't fail because you may want to transfer
# a log file that possibly MAY exist but keep going to fetch other
# log files. Today, this is better achieved by adding
# ignore_errors or failed_when to the task. Control the behaviour
# via fail_when_missing
if fail_on_missing:
result['failed'] = True
del result['changed']
else:
result['msg'] += ", not transferring, ignored"
return result
dest = os.path.normpath(dest)
# calculate checksum for the local file
local_checksum = checksum(dest)
......@@ -132,7 +169,15 @@ class ActionModule(ActionBase):
makedirs_safe(os.path.dirname(dest))
# fetch the file and check for changes
self._connection.fetch_file(source, dest)
if remote_data is None:
self._connection.fetch_file(source, dest)
else:
try:
f = open(to_bytes(dest, errors='surrogate_or_strict'), 'wb')
f.write(remote_data)
f.close()
except (IOError, OSError) as e:
raise AnsibleActionFail("Failed to fetch the file: %s" % e)
new_checksum = secure_hash(dest)
# For backwards compatibility. We'll return None on FIPS enabled systems
try:
......
mitogen-0.2.9/ansible_mitogen/plugins/action/mitogen_get_stack.py → mitogen-0.3.7/ansible_mitogen/plugins/action/mitogen_get_stack.py
View file @ 985463fd
......@@ -26,14 +26,15 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import unicode_literals
"""
Fetch the connection configuration stack that would be used to connect to a
target, without actually connecting to it.
"""
from __future__ import absolute_import, division, print_function
from __future__ import unicode_literals
__metaclass__ = type
import ansible_mitogen.connection
from ansible.plugins.action import ActionBase
......@@ -52,4 +53,6 @@ class ActionModule(ActionBase):
'changed': True,
'result': stack,
'_ansible_verbose_always': True,
# for ansible < 2.8, we'll default to /usr/bin/python like before
'discovered_interpreter': self._connection._action._discovered_interpreter
}
mitogen-0.2.9/ansible_mitogen/plugins/connection/mitogen_buildah.py → mitogen-0.3.7/ansible_mitogen/plugins/connection/mitogen_buildah.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os.path
import sys
......
mitogen-0.2.9/ansible_mitogen/plugins/connection/mitogen_doas.py → mitogen-0.3.7/ansible_mitogen/plugins/connection/mitogen_doas.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os.path
import sys
......
mitogen-0.2.9/ansible_mitogen/plugins/connection/mitogen_docker.py → mitogen-0.3.7/ansible_mitogen/plugins/connection/mitogen_docker.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os.path
import sys
......
mitogen-0.2.9/ansible_mitogen/plugins/connection/mitogen_jail.py → mitogen-0.3.7/ansible_mitogen/plugins/connection/mitogen_jail.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os.path
import sys
......
mitogen-0.2.9/ansible_mitogen/plugins/connection/mitogen_kubectl.py → mitogen-0.3.7/ansible_mitogen/plugins/connection/mitogen_kubectl.py
View file @ 985463fd
......@@ -27,12 +27,13 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os.path
import sys
from ansible.errors import AnsibleConnectionFailure
from ansible.module_utils.six import iteritems
import ansible.errors
try:
import ansible_mitogen
......@@ -45,17 +46,11 @@ import ansible_mitogen.connection
import ansible_mitogen.loaders
_class = ansible_mitogen.loaders.connection_loader__get(
_get_result = ansible_mitogen.loaders.connection_loader__get(
'kubectl',
class_only=True,
)
if _class:
kubectl = sys.modules[_class.__module__]
del _class
else:
kubectl = None
class Connection(ansible_mitogen.connection.Connection):
transport = 'kubectl'
......@@ -66,14 +61,22 @@ class Connection(ansible_mitogen.connection.Connection):
)
def __init__(self, *args, **kwargs):
if kubectl is None:
raise AnsibleConnectionFailure(self.not_supported_msg)
if not _get_result:
raise ansible.errors.AnsibleConnectionFailure(self.not_supported_msg)
super(Connection, self).__init__(*args, **kwargs)
def get_extra_args(self):
try:
# Ansible < 2.10, _get_result is the connection class
connection_options = _get_result.connection_options
except AttributeError:
# Ansible >= 2.10, _get_result is a get_with_context_result
connection_options = _get_result.object.connection_options
parameters = []
for key, option in iteritems(kubectl.CONNECTION_OPTIONS):
if self.get_task_var('ansible_' + key) is not None:
parameters += [ option, self.get_task_var('ansible_' + key) ]
for key in connection_options:
task_var_name = 'ansible_%s' % key
task_var = self.get_task_var(task_var_name)
if task_var is not None:
parameters += [connection_options[key], task_var]
return parameters
mitogen-0.2.9/ansible_mitogen/plugins/connection/mitogen_local.py → mitogen-0.3.7/ansible_mitogen/plugins/connection/mitogen_local.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os.path
import sys
......
mitogen-0.2.9/ansible_mitogen/plugins/connection/mitogen_lxc.py → mitogen-0.3.7/ansible_mitogen/plugins/connection/mitogen_lxc.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os.path
import sys
......
mitogen-0.2.9/ansible_mitogen/plugins/connection/mitogen_lxd.py → mitogen-0.3.7/ansible_mitogen/plugins/connection/mitogen_lxd.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os.path
import sys
......
mitogen-0.2.9/ansible_mitogen/plugins/connection/mitogen_machinectl.py → mitogen-0.3.7/ansible_mitogen/plugins/connection/mitogen_machinectl.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os.path
import sys
......
mitogen-0.3.7/ansible_mitogen/plugins/connection/mitogen_podman.py 0 → 100644
View file @ 985463fd
# Copyright 2022, Mitogen contributers
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
# this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright notice,
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# 3. Neither the name of the copyright holder nor the names of its contributors
# may be used to endorse or promote products derived from this software without
# specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os.path
import sys
try:
import ansible_mitogen
except ImportError:
base_dir = os.path.dirname(__file__)
sys.path.insert(0, os.path.abspath(os.path.join(base_dir, '../../..')))
del base_dir
import ansible_mitogen.connection
class Connection(ansible_mitogen.connection.Connection):
transport = 'podman'
mitogen-0.2.9/ansible_mitogen/plugins/connection/mitogen_setns.py → mitogen-0.3.7/ansible_mitogen/plugins/connection/mitogen_setns.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os.path
import sys
......
mitogen-0.2.9/ansible_mitogen/plugins/connection/mitogen_ssh.py → mitogen-0.3.7/ansible_mitogen/plugins/connection/mitogen_ssh.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os.path
import sys
......@@ -40,6 +42,24 @@ DOCUMENTATION = """
accepts.
version_added: "2.5"
options:
ssh_args:
type: str
vars:
- name: ssh_args
- name: ansible_ssh_args
- name: ansible_mitogen_ssh_args
ssh_common_args:
type: str
vars:
- name: ssh_args
- name: ansible_ssh_common_args
- name: ansible_mitogen_ssh_common_args
ssh_extra_args:
type: str
vars:
- name: ssh_args
- name: ansible_ssh_extra_args
- name: ansible_mitogen_ssh_extra_args
"""
try:
......
mitogen-0.2.9/ansible_mitogen/plugins/connection/mitogen_su.py → mitogen-0.3.7/ansible_mitogen/plugins/connection/mitogen_su.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os.path
import sys
......
mitogen-0.2.9/ansible_mitogen/plugins/connection/mitogen_sudo.py → mitogen-0.3.7/ansible_mitogen/plugins/connection/mitogen_sudo.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os.path
import sys
......
mitogen-0.2.9/ansible_mitogen/plugins/strategy/mitogen.py → mitogen-0.3.7/ansible_mitogen/plugins/strategy/mitogen.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os.path
import sys
......
mitogen-0.2.9/ansible_mitogen/plugins/strategy/mitogen_free.py → mitogen-0.3.7/ansible_mitogen/plugins/strategy/mitogen_free.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os.path
import sys
......
mitogen-0.2.9/ansible_mitogen/plugins/strategy/mitogen_host_pinned.py → mitogen-0.3.7/ansible_mitogen/plugins/strategy/mitogen_host_pinned.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os.path
import sys
......
mitogen-0.2.9/ansible_mitogen/plugins/strategy/mitogen_linear.py → mitogen-0.3.7/ansible_mitogen/plugins/strategy/mitogen_linear.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os.path
import sys
......
mitogen-0.2.9/ansible_mitogen/process.py → mitogen-0.3.7/ansible_mitogen/process.py
View file @ 985463fd
......@@ -26,7 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import atexit
import logging
import multiprocessing
......@@ -178,42 +180,6 @@ def setup_pool(pool):
LOG.debug('Service pool configured: size=%d', pool.size)
def _setup_simplejson(responder):
"""
We support serving simplejson for Python 2.4 targets on Ansible 2.3, at
least so the package's own CI Docker scripts can run without external
help, however newer versions of simplejson no longer support Python
2.4. Therefore override any installed/loaded version with a
2.4-compatible version we ship in the compat/ directory.
"""
responder.whitelist_prefix('simplejson')
# issue #536: must be at end of sys.path, in case existing newer
# version is already loaded.
compat_path = os.path.join(os.path.dirname(__file__), 'compat')
sys.path.append(compat_path)
for fullname, is_pkg, suffix in (
(u'simplejson', True, '__init__.py'),
(u'simplejson.decoder', False, 'decoder.py'),
(u'simplejson.encoder', False, 'encoder.py'),
(u'simplejson.scanner', False, 'scanner.py'),
):
path = os.path.join(compat_path, 'simplejson', suffix)
fp = open(path, 'rb')
try:
source = fp.read()
finally:
fp.close()
responder.add_source_override(
fullname=fullname,
path=path,
source=source,
is_pkg=is_pkg,
)
def _setup_responder(responder):
"""
Configure :class:`mitogen.master.ModuleResponder` to only permit
......@@ -221,7 +187,6 @@ def _setup_responder(responder):
"""
responder.whitelist_prefix('ansible')
responder.whitelist_prefix('ansible_mitogen')
_setup_simplejson(responder)
# Ansible 2.3 is compatible with Python 2.4 targets, however
# ansible/__init__.py is not. Instead, executor/module_common.py writes
......
mitogen-0.2.9/ansible_mitogen/runner.py → mitogen-0.3.7/ansible_mitogen/runner.py
View file @ 985463fd
......@@ -36,8 +36,11 @@ Each class in here has a corresponding Planner class in planners.py that knows
how to build arguments for it, preseed related data, etc.
"""
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import atexit
import imp
import json
import os
import re
import shlex
......@@ -61,10 +64,12 @@ except ImportError:
ctypes = None
try:
import json
# Python >= 3.4, PEP 451 ModuleSpec API
import importlib.machinery
import importlib.util
except ImportError:
# Python 2.4
import simplejson as json
# Python < 3.4, PEP 302 Import Hooks
import imp
try:
# Cannot use cStringIO as it does not support Unicode.
......@@ -516,10 +521,71 @@ class ModuleUtilsImporter(object):
sys.modules.pop(fullname, None)
def find_module(self, fullname, path=None):
"""
Return a loader for the module with fullname, if we will load it.
Implements importlib.abc.MetaPathFinder.find_module().
Deprecrated in Python 3.4+, replaced by find_spec().
Raises ImportWarning in Python 3.10+. Removed in Python 3.12.
"""
if fullname in self._by_fullname:
return self
def find_spec(self, fullname, path, target=None):
"""
Return a `ModuleSpec` for module with `fullname` if we will load it.
Otherwise return `None`.
Implements importlib.abc.MetaPathFinder.find_spec(). Python 3.4+.
"""
if fullname.endswith('.'):
return None
try:
module_path, is_package = self._by_fullname[fullname]
except KeyError:
LOG.debug('Skipping %s: not present', fullname)
return None
LOG.debug('Handling %s', fullname)
origin = 'master:%s' % (module_path,)
return importlib.machinery.ModuleSpec(
fullname, loader=self, origin=origin, is_package=is_package,
)
def create_module(self, spec):
"""
Return a module object for the given ModuleSpec.
Implements PEP-451 importlib.abc.Loader API introduced in Python 3.4.
Unlike Loader.load_module() this shouldn't populate sys.modules or
set module attributes. Both are done by Python.
"""
module = types.ModuleType(spec.name)
# FIXME create_module() shouldn't initialise module attributes
module.__file__ = spec.origin
return module
def exec_module(self, module):
"""
Execute the module to initialise it. Don't return anything.
Implements PEP-451 importlib.abc.Loader API, introduced in Python 3.4.
"""
spec = module.__spec__
path, _ = self._by_fullname[spec.name]
source = ansible_mitogen.target.get_small_file(self._context, path)
code = compile(source, path, 'exec', 0, 1)
exec(code, module.__dict__)
self._loaded.add(spec.name)
def load_module(self, fullname):
"""
Return the loaded module specified by fullname.
Implements PEP 302 importlib.abc.Loader.load_module().
Deprecated in Python 3.4+, replaced by create_module() & exec_module().
"""
path, is_pkg = self._by_fullname[fullname]
source = ansible_mitogen.target.get_small_file(self._context, path)
code = compile(source, path, 'exec', 0, 1)
......@@ -820,12 +886,17 @@ class NewStyleRunner(ScriptRunner):
synchronization mechanism by importing everything the module will need
prior to detaching.
"""
# I think "custom" means "found in custom module_utils search path",
# e.g. playbook relative dir, ~/.ansible/..., Ansible collection.
for fullname, _, _ in self.module_map['custom']:
mitogen.core.import_module(fullname)
# I think "builtin" means "part of ansible/ansible-base/ansible-core",
# as opposed to Python builtin modules such as sys.
for fullname in self.module_map['builtin']:
try:
mitogen.core.import_module(fullname)
except ImportError:
except ImportError as exc:
# #590: Ansible 2.8 module_utils.distro is a package that
# replaces itself in sys.modules with a non-package during
# import. Prior to replacement, it is a real package containing
......@@ -836,8 +907,18 @@ class NewStyleRunner(ScriptRunner):
# loop progresses to the next entry and attempts to preload
# 'distro._distro', the import mechanism will fail. So here we
# silently ignore any failure for it.
if fullname != 'ansible.module_utils.distro._distro':
raise
if fullname == 'ansible.module_utils.distro._distro':
continue
# ansible.module_utils.compat.selinux raises ImportError if it
# can't load libselinux.so. The importer would usually catch
# this & skip selinux operations. We don't care about selinux,
# we're using import to get a copy of the module.
if (fullname == 'ansible.module_utils.compat.selinux'
and exc.msg == 'unable to load libselinux.so'):
continue
raise
def _setup_excepthook(self):
"""
......
mitogen-0.2.9/ansible_mitogen/services.py → mitogen-0.3.7/ansible_mitogen/services.py
View file @ 985463fd
......@@ -39,23 +39,23 @@ connections, grant access to files by children, and register for notification
when a child has completed a job.
"""
from __future__ import absolute_import
from __future__ import absolute_import, division, print_function
from __future__ import unicode_literals
__metaclass__ = type
import logging
import os
import os.path
import sys
import threading
import ansible.constants
import mitogen
import mitogen.core
import mitogen.service
import mitogen.utils
import ansible_mitogen.loaders
import ansible_mitogen.module_finder
import ansible_mitogen.target
import ansible_mitogen.utils.unsafe
LOG = logging.getLogger(__name__)
......@@ -91,7 +91,7 @@ def _get_candidate_temp_dirs():
remote_tmp = ansible.constants.DEFAULT_REMOTE_TMP
system_tmpdirs = ('/var/tmp', '/tmp')
return mitogen.utils.cast([remote_tmp] + list(system_tmpdirs))
return ansible_mitogen.utils.unsafe.cast([remote_tmp] + list(system_tmpdirs))
def key_from_dict(**kwargs):
......@@ -170,6 +170,12 @@ class ContextService(mitogen.service.Service):
"""
LOG.debug('%r.reset(%r)', self, stack)
# this could happen if we have a `shutdown -r` shell command
# and then a `wait_for_connection` right afterwards
# in this case, we have no stack to disconnect from
if not stack:
return False
l = mitogen.core.Latch()
context = None
with self._lock:
......
mitogen-0.2.9/ansible_mitogen/strategy.py → mitogen-0.3.7/ansible_mitogen/strategy.py
View file @ 985463fd
......@@ -26,8 +26,9 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
import distutils.version
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os
import signal
import threading
......@@ -43,52 +44,8 @@ import ansible_mitogen.loaders
import ansible_mitogen.mixins
import ansible_mitogen.process
import ansible
import ansible.executor.process.worker
try:
# 2.8+ has a standardized "unset" object.
from ansible.utils.sentinel import Sentinel
except ImportError:
Sentinel = None
ANSIBLE_VERSION_MIN = (2, 3)
ANSIBLE_VERSION_MAX = (2, 9)
NEW_VERSION_MSG = (
"Your Ansible version (%s) is too recent. The most recent version\n"
"supported by Mitogen for Ansible is %s.x. Please check the Mitogen\n"
"release notes to see if a new version is available, otherwise\n"
"subscribe to the corresponding GitHub issue to be notified when\n"
"support becomes available.\n"
"\n"
" https://mitogen.rtfd.io/en/latest/changelog.html\n"
" https://github.com/dw/mitogen/issues/\n"
)
OLD_VERSION_MSG = (
"Your version of Ansible (%s) is too old. The oldest version supported by "
"Mitogen for Ansible is %s."
)
def _assert_supported_release():
"""
Throw AnsibleError with a descriptive message in case of being loaded into
an unsupported Ansible release.
"""
v = ansible.__version__
if not isinstance(v, tuple):
v = tuple(distutils.version.LooseVersion(v).version)
if v[:2] < ANSIBLE_VERSION_MIN:
raise ansible.errors.AnsibleError(
OLD_VERSION_MSG % (v, ANSIBLE_VERSION_MIN)
)
if v[:2] > ANSIBLE_VERSION_MAX:
raise ansible.errors.AnsibleError(
NEW_VERSION_MSG % (ansible.__version__, ANSIBLE_VERSION_MAX)
)
import ansible.utils.sentinel
def _patch_awx_callback():
......@@ -99,12 +56,11 @@ def _patch_awx_callback():
# AWX uses sitecustomize.py to force-load this package. If it exists, we're
# running under AWX.
try:
from awx_display_callback.events import EventContext
from awx_display_callback.events import event_context
import awx_display_callback.events
except ImportError:
return
if hasattr(EventContext(), '_local'):
if hasattr(awx_display_callback.events.EventContext(), '_local'):
# Patched version.
return
......@@ -113,8 +69,8 @@ def _patch_awx_callback():
ctx = tls.setdefault('_ctx', {})
ctx.update(kwargs)
EventContext._local = threading.local()
EventContext.add_local = patch_add_local
awx_display_callback.events.EventContext._local = threading.local()
awx_display_callback.events.EventContext.add_local = patch_add_local
_patch_awx_callback()
......@@ -132,8 +88,7 @@ def wrap_action_loader__get(name, *args, **kwargs):
get_kwargs = {'class_only': True}
if name in ('fetch',):
name = 'mitogen_' + name
if ansible.__version__ >= '2.8':
get_kwargs['collection_list'] = kwargs.pop('collection_list', None)
get_kwargs['collection_list'] = kwargs.pop('collection_list', None)
klass = ansible_mitogen.loaders.action_loader__get(name, **get_kwargs)
if klass:
......@@ -153,6 +108,7 @@ REDIRECTED_CONNECTION_PLUGINS = (
'lxc',
'lxd',
'machinectl',
'podman',
'setns',
'ssh',
)
......@@ -217,7 +173,7 @@ class AnsibleWrappers(object):
with references to the real functions.
"""
ansible_mitogen.loaders.action_loader.get = wrap_action_loader__get
ansible_mitogen.loaders.connection_loader.get = wrap_connection_loader__get
ansible_mitogen.loaders.connection_loader.get_with_context = wrap_connection_loader__get
global worker__run
worker__run = ansible.executor.process.worker.WorkerProcess.run
......@@ -230,7 +186,7 @@ class AnsibleWrappers(object):
ansible_mitogen.loaders.action_loader.get = (
ansible_mitogen.loaders.action_loader__get
)
ansible_mitogen.loaders.connection_loader.get = (
ansible_mitogen.loaders.connection_loader.get_with_context = (
ansible_mitogen.loaders.connection_loader__get
)
ansible.executor.process.worker.WorkerProcess.run = worker__run
......@@ -324,7 +280,7 @@ class StrategyMixin(object):
name=task.action,
class_only=True,
)
if play_context.connection is not Sentinel:
if play_context.connection is not ansible.utils.sentinel.Sentinel:
# 2.8 appears to defer computing this until inside the worker.
# TODO: figure out where it has moved.
ansible_mitogen.loaders.connection_loader.get(
......@@ -352,7 +308,6 @@ class StrategyMixin(object):
Wrap :meth:`run` to ensure requisite infrastructure and modifications
are configured for the duration of the call.
"""
_assert_supported_release()
wrappers = AnsibleWrappers()
self._worker_model = self._get_worker_model()
ansible_mitogen.process.set_worker_model(self._worker_model)
......
mitogen-0.2.9/ansible_mitogen/target.py → mitogen-0.3.7/ansible_mitogen/target.py
View file @ 985463fd
......@@ -33,8 +33,12 @@ Helper functions intended to be executed on the target. These are entrypoints
for file transfer, module execution and sundry bits like changing file modes.
"""
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import errno
import grp
import json
import operator
import os
import pwd
......@@ -51,16 +55,10 @@ import types
logging = __import__('logging')
import mitogen.core
import mitogen.fork
import mitogen.parent
import mitogen.service
from mitogen.core import b
try:
import json
except ImportError:
import simplejson as json
try:
reduce
except NameError:
......@@ -144,7 +142,7 @@ def subprocess__Popen__close_fds(self, but):
if (
sys.platform.startswith(u'linux') and
sys.version < u'3.0' and
sys.version_info < (3,) and
hasattr(subprocess.Popen, u'_close_fds') and
not mitogen.is_master
):
......@@ -369,11 +367,6 @@ def init_child(econtext, log_level, candidate_temp_dirs):
LOG.setLevel(log_level)
logging.getLogger('ansible_mitogen').setLevel(log_level)
# issue #536: if the json module is available, remove simplejson from the
# importer whitelist to avoid confusing certain Ansible modules.
if json.__name__ == 'json':
econtext.importer.whitelist.remove('simplejson')
global _fork_parent
if FORK_SUPPORTED:
mitogen.parent.upgrade_router(econtext)
......@@ -652,7 +645,8 @@ def read_path(path):
"""
Fetch the contents of a filesystem `path` as bytes.
"""
return open(path, 'rb').read()
with open(path, 'rb') as f:
return f.read()
def set_file_owner(path, owner, group=None, fd=None):
......
mitogen-0.2.9/ansible_mitogen/transport_config.py → mitogen-0.3.7/ansible_mitogen/transport_config.py
View file @ 985463fd
......@@ -26,9 +26,6 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
from __future__ import absolute_import
from __future__ import unicode_literals
"""
Mitogen extends Ansible's target configuration mechanism in several ways that
require some care:
......@@ -60,6 +57,10 @@ information from PlayContext, and another that takes (almost) all information
from HostVars.
"""
from __future__ import absolute_import, division, print_function
from __future__ import unicode_literals
__metaclass__ = type
import abc
import os
import ansible.utils.shlex
......@@ -67,17 +68,90 @@ import ansible.constants as C
from ansible.module_utils.six import with_metaclass
# this was added in Ansible >= 2.8.0; fallback to the default interpreter if necessary
try:
from ansible.executor.interpreter_discovery import discover_interpreter
except ImportError:
discover_interpreter = lambda action,interpreter_name,discovery_mode,task_vars: '/usr/bin/python'
try:
from ansible.utils.unsafe_proxy import AnsibleUnsafeText
except ImportError:
from ansible.vars.unsafe_proxy import AnsibleUnsafeText
import ansible_mitogen.loaders
import mitogen.core
def parse_python_path(s):
def run_interpreter_discovery_if_necessary(s, task_vars, action, rediscover_python):
"""
Triggers ansible python interpreter discovery if requested.
Caches this value the same way Ansible does it.
For connections like `docker`, we want to rediscover the python interpreter because
it could be different than what's ran on the host
"""
# keep trying different interpreters until we don't error
if action._finding_python_interpreter:
return action._possible_python_interpreter
if s in ['auto', 'auto_legacy', 'auto_silent', 'auto_legacy_silent']:
# python is the only supported interpreter_name as of Ansible 2.8.8
interpreter_name = 'python'
discovered_interpreter_config = u'discovered_interpreter_%s' % interpreter_name
if task_vars.get('ansible_facts') is None:
task_vars['ansible_facts'] = {}
if rediscover_python and task_vars.get('ansible_facts', {}).get(discovered_interpreter_config):
# if we're rediscovering python then chances are we're running something like a docker connection
# this will handle scenarios like running a playbook that does stuff + then dynamically creates a docker container,
# then runs the rest of the playbook inside that container, and then rerunning the playbook again
action._rediscovered_python = True
# blow away the discovered_interpreter_config cache and rediscover
del task_vars['ansible_facts'][discovered_interpreter_config]
if discovered_interpreter_config not in task_vars['ansible_facts']:
action._finding_python_interpreter = True
# fake pipelining so discover_interpreter can be happy
action._connection.has_pipelining = True
s = AnsibleUnsafeText(discover_interpreter(
action=action,
interpreter_name=interpreter_name,
discovery_mode=s,
task_vars=task_vars))
# cache discovered interpreter
task_vars['ansible_facts'][discovered_interpreter_config] = s
action._connection.has_pipelining = False
else:
s = task_vars['ansible_facts'][discovered_interpreter_config]
# propagate discovered interpreter as fact
action._discovered_interpreter_key = discovered_interpreter_config
action._discovered_interpreter = s
action._finding_python_interpreter = False
return s
def parse_python_path(s, task_vars, action, rediscover_python):
"""
Given the string set for ansible_python_interpeter, parse it using shell
syntax and return an appropriate argument vector.
syntax and return an appropriate argument vector. If the value detected is
one of interpreter discovery then run that first. Caches python interpreter
discovery value in `facts_from_task_vars` like how Ansible handles this.
"""
if s:
return ansible.utils.shlex.shlex_split(s)
if not s:
# if python_path doesn't exist, default to `auto` and attempt to discover it
s = 'auto'
s = run_interpreter_discovery_if_necessary(s, task_vars, action, rediscover_python)
# if unable to determine python_path, fallback to '/usr/bin/python'
if not s:
s = '/usr/bin/python'
return ansible.utils.shlex.shlex_split(s)
def optional_secret(value):
......@@ -282,6 +356,12 @@ class Spec(with_metaclass(abc.ABCMeta, object)):
The path to the "machinectl" program for the 'setns' transport.
"""
@abc.abstractmethod
def mitogen_podman_path(self):
"""
The path to the "podman" program for the 'podman' transport.
"""
@abc.abstractmethod
def mitogen_ssh_keepalive_interval(self):
"""
......@@ -330,6 +410,9 @@ class PlayContextSpec(Spec):
self._play_context = play_context
self._transport = transport
self._inventory_name = inventory_name
self._task_vars = self._connection._get_task_vars()
# used to run interpreter discovery
self._action = connection._action
def transport(self):
return self._transport
......@@ -353,7 +436,10 @@ class PlayContextSpec(Spec):
return self._play_context.become_user
def become_pass(self):
return optional_secret(self._play_context.become_pass)
become_method = self.become_method()
become_plugin = ansible_mitogen.loaders.become_loader.get(become_method)
become_pass = become_plugin.get_option('become_pass', hostvars=self._task_vars)
return optional_secret(become_pass)
def password(self):
return optional_secret(self._play_context.password)
......@@ -361,18 +447,22 @@ class PlayContextSpec(Spec):
def port(self):
return self._play_context.port
def python_path(self):
def python_path(self, rediscover_python=False):
s = self._connection.get_task_var('ansible_python_interpreter')
# #511, #536: executor/module_common.py::_get_shebang() hard-wires
# "/usr/bin/python" as the default interpreter path if no other
# interpreter is specified.
return parse_python_path(s or '/usr/bin/python')
return parse_python_path(
s,
task_vars=self._task_vars,
action=self._action,
rediscover_python=rediscover_python)
def private_key_file(self):
return self._play_context.private_key_file
def ssh_executable(self):
return self._play_context.ssh_executable
return C.config.get_config_value("ssh_executable", plugin_type="connection", plugin_name="ssh", variables=self._task_vars.get("vars", {}))
def timeout(self):
return self._play_context.timeout
......@@ -388,9 +478,9 @@ class PlayContextSpec(Spec):
return [
mitogen.core.to_text(term)
for s in (
getattr(self._play_context, 'ssh_args', ''),
getattr(self._play_context, 'ssh_common_args', ''),
getattr(self._play_context, 'ssh_extra_args', '')
C.config.get_config_value("ssh_args", plugin_type="connection", plugin_name="ssh", variables=self._task_vars.get("vars", {})),
C.config.get_config_value("ssh_common_args", plugin_type="connection", plugin_name="ssh", variables=self._task_vars.get("vars", {})),
C.config.get_config_value("ssh_extra_args", plugin_type="connection", plugin_name="ssh", variables=self._task_vars.get("vars", {}))
)
for term in ansible.utils.shlex.shlex_split(s or '')
]
......@@ -448,6 +538,9 @@ class PlayContextSpec(Spec):
def mitogen_lxc_info_path(self):
return self._connection.get_task_var('mitogen_lxc_info_path')
def mitogen_podman_path(self):
return self._connection.get_task_var('mitogen_podman_path')
def mitogen_ssh_keepalive_interval(self):
return self._connection.get_task_var('mitogen_ssh_keepalive_interval')
......@@ -490,14 +583,16 @@ class MitogenViaSpec(Spec):
having a configruation problem with connection delegation, the answer to
your problem lies in the method implementations below!
"""
def __init__(self, inventory_name, host_vars, become_method, become_user,
play_context):
def __init__(self, inventory_name, host_vars, task_vars, become_method, become_user,
play_context, action):
"""
:param str inventory_name:
The inventory name of the intermediary machine, i.e. not the target
machine.
:param dict host_vars:
The HostVars magic dictionary provided by Ansible in task_vars.
:param dict task_vars:
Task vars provided by Ansible.
:param str become_method:
If the mitogen_via= spec included a become method, the method it
specifies.
......@@ -509,14 +604,18 @@ class MitogenViaSpec(Spec):
the real target machine. Values from this object are **strictly
restricted** to values that are Ansible-global, e.g. the passwords
specified interactively.
:param ActionModuleMixin action:
Backref to the ActionModuleMixin required for ansible interpreter discovery
"""
self._inventory_name = inventory_name
self._host_vars = host_vars
self._task_vars = task_vars
self._become_method = become_method
self._become_user = become_user
# Dangerous! You may find a variable you want in this object, but it's
# almost certainly for the wrong machine!
self._dangerous_play_context = play_context
self._action = action
def transport(self):
return (
......@@ -557,8 +656,8 @@ class MitogenViaSpec(Spec):
def become_pass(self):
return optional_secret(
self._host_vars.get('ansible_become_password') or
self._host_vars.get('ansible_become_pass')
self._host_vars.get('ansible_become_pass') or
self._host_vars.get('ansible_become_password')
)
def password(self):
......@@ -574,12 +673,16 @@ class MitogenViaSpec(Spec):
C.DEFAULT_REMOTE_PORT
)
def python_path(self):
def python_path(self, rediscover_python=False):
s = self._host_vars.get('ansible_python_interpreter')
# #511, #536: executor/module_common.py::_get_shebang() hard-wires
# "/usr/bin/python" as the default interpreter path if no other
# interpreter is specified.
return parse_python_path(s or '/usr/bin/python')
return parse_python_path(
s,
task_vars=self._task_vars,
action=self._action,
rediscover_python=rediscover_python)
def private_key_file(self):
# TODO: must come from PlayContext too.
......@@ -590,10 +693,7 @@ class MitogenViaSpec(Spec):
)
def ssh_executable(self):
return (
self._host_vars.get('ansible_ssh_executable') or
C.ANSIBLE_SSH_EXECUTABLE
)
return C.config.get_config_value("ssh_executable", plugin_type="connection", plugin_name="ssh", variables=self._task_vars.get("vars", {}))
def timeout(self):
# TODO: must come from PlayContext too.
......@@ -610,22 +710,9 @@ class MitogenViaSpec(Spec):
return [
mitogen.core.to_text(term)
for s in (
(
self._host_vars.get('ansible_ssh_args') or
getattr(C, 'ANSIBLE_SSH_ARGS', None) or
os.environ.get('ANSIBLE_SSH_ARGS')
# TODO: ini entry. older versions.
),
(
self._host_vars.get('ansible_ssh_common_args') or
os.environ.get('ANSIBLE_SSH_COMMON_ARGS')
# TODO: ini entry.
),
(
self._host_vars.get('ansible_ssh_extra_args') or
os.environ.get('ANSIBLE_SSH_EXTRA_ARGS')
# TODO: ini entry.
),
C.config.get_config_value("ssh_args", plugin_type="connection", plugin_name="ssh", variables=self._task_vars.get("vars", {})),
C.config.get_config_value("ssh_common_args", plugin_type="connection", plugin_name="ssh", variables=self._task_vars.get("vars", {})),
C.config.get_config_value("ssh_extra_args", plugin_type="connection", plugin_name="ssh", variables=self._task_vars.get("vars", {}))
)
for term in ansible.utils.shlex.shlex_split(s)
if s
......@@ -666,7 +753,7 @@ class MitogenViaSpec(Spec):
return self._host_vars.get('mitogen_kubectl_path')
def mitogen_lxc_path(self):
return self.host_vars.get('mitogen_lxc_path')
return self._host_vars.get('mitogen_lxc_path')
def mitogen_lxc_attach_path(self):
return self._host_vars.get('mitogen_lxc_attach_path')
......@@ -674,6 +761,9 @@ class MitogenViaSpec(Spec):
def mitogen_lxc_info_path(self):
return self._host_vars.get('mitogen_lxc_info_path')
def mitogen_podman_path(self):
return self._host_vars.get('mitogen_podman_path')
def mitogen_ssh_keepalive_interval(self):
return self._host_vars.get('mitogen_ssh_keepalive_interval')
......
mitogen-0.3.7/ansible_mitogen/utils/__init__.py 0 → 100644
View file @ 985463fd
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import re
import ansible
__all__ = [
'ansible_version',
]
def _parse(v_string):
# Adapted from distutils.version.LooseVersion.parse()
component_re = re.compile(r'(\d+ | [a-z]+ | \.)', re.VERBOSE)
for component in component_re.split(v_string):
if not component or component == '.':
continue
try:
yield int(component)
except ValueError:
yield component
ansible_version = tuple(_parse(ansible.__version__))
del _parse
del re
del ansible
mitogen-0.3.7/ansible_mitogen/utils/unsafe.py 0 → 100644
View file @ 985463fd
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import ansible
import ansible.utils.unsafe_proxy
import ansible_mitogen.utils
import mitogen
import mitogen.core
import mitogen.utils
__all__ = [
'cast',
]
def _cast_to_dict(obj): return {cast(k): cast(v) for k, v in obj.items()}
def _cast_to_list(obj): return [cast(v) for v in obj]
def _cast_unsafe(obj): return obj._strip_unsafe()
def _passthrough(obj): return obj
# A dispatch table to cast objects based on their exact type.
# This is an optimisation, reliable fallbacks are required (e.g. isinstance())
_CAST_DISPATCH = {
bytes: bytes,
dict: _cast_to_dict,
list: _cast_to_list,
tuple: _cast_to_list,
mitogen.core.UnicodeType: mitogen.core.UnicodeType,
}
_CAST_DISPATCH.update({t: _passthrough for t in mitogen.utils.PASSTHROUGH})
if hasattr(ansible.utils.unsafe_proxy.AnsibleUnsafeText, '_strip_unsafe'):
_CAST_DISPATCH.update({
ansible.utils.unsafe_proxy.AnsibleUnsafeBytes: _cast_unsafe,
ansible.utils.unsafe_proxy.AnsibleUnsafeText: _cast_unsafe,
ansible.utils.unsafe_proxy.NativeJinjaUnsafeText: _cast_unsafe,
})
elif ansible_mitogen.utils.ansible_version[:2] <= (2, 16):
_CAST_DISPATCH.update({
ansible.utils.unsafe_proxy.AnsibleUnsafeBytes: bytes,
ansible.utils.unsafe_proxy.AnsibleUnsafeText: mitogen.core.UnicodeType,
})
else:
mitogen_ver = '.'.join(str(v) for v in mitogen.__version__)
raise ImportError("Mitogen %s can't unwrap Ansible %s AnsibleUnsafe objects"
% (mitogen_ver, ansible.__version__))
def cast(obj):
"""
Return obj (or a copy) with subtypes of builtins cast to their supertype.
This is an enhanced version of :func:`mitogen.utils.cast`. In addition it
handles ``ansible.utils.unsafe_proxy.AnsibleUnsafeText`` and variants.
There are types handled by :func:`ansible.utils.unsafe_proxy.wrap_var()`
that this function currently does not handle (e.g. `set()`), or preserve
preserve (e.g. `tuple()`). Future enhancements may change this.
:param obj:
Object to undecorate.
:returns:
Undecorated object.
"""
# Fast path: obj is a known type, dispatch directly
try:
unwrapper = _CAST_DISPATCH[type(obj)]
except KeyError:
pass
else:
return unwrapper(obj)
# Slow path: obj is some unknown subclass
if isinstance(obj, dict): return _cast_to_dict(obj)
if isinstance(obj, (list, tuple)): return _cast_to_list(obj)
return mitogen.utils.cast(obj)
mitogen-0.2.9/PKG-INFO → mitogen-0.3.7/mitogen.egg-info/PKG-INFO
View file @ 985463fd
Metadata-Version: 1.1
Metadata-Version: 2.1
Name: mitogen
Version: 0.2.9
Version: 0.3.7
Summary: Library for writing distributed self-replicating programs.
Home-page: https://github.com/dw/mitogen/
Home-page: https://github.com/mitogen-hq/mitogen/
Author: David Wilson
Author-email: UNKNOWN
License: New BSD
Description: UNKNOWN
Platform: UNKNOWN
Classifier: Environment :: Console
Classifier: Framework :: Ansible
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2.4
Classifier: Programming Language :: Python :: 2.5
Classifier: Programming Language :: Python :: 2.6
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: System :: Distributed Computing
Classifier: Topic :: System :: Systems Administration
Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*
Description-Content-Type: text/markdown
License-File: LICENSE
# Mitogen
<a href="https://mitogen.networkgenomics.com/">Please see the documentation</a>.
![](https://i.imgur.com/eBM6LhJ.gif)
[![Total alerts](https://img.shields.io/lgtm/alerts/g/mitogen-hq/mitogen.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/mitogen-hq/mitogen/alerts/)
[![Build Status](https://dev.azure.com/mitogen-hq/mitogen/_apis/build/status/mitogen-hq.mitogen?branchName=master)](https://dev.azure.com/mitogen-hq/mitogen/_build/latest?definitionId=1&branchName=master)
mitogen-0.2.9/mitogen.egg-info/SOURCES.txt → mitogen-0.3.7/mitogen.egg-info/SOURCES.txt
View file @ 985463fd
......@@ -19,10 +19,6 @@ ansible_mitogen/strategy.py
ansible_mitogen/target.py
ansible_mitogen/transport_config.py
ansible_mitogen/compat/__init__.py
ansible_mitogen/compat/simplejson/__init__.py
ansible_mitogen/compat/simplejson/decoder.py
ansible_mitogen/compat/simplejson/encoder.py
ansible_mitogen/compat/simplejson/scanner.py
ansible_mitogen/plugins/__init__.py
ansible_mitogen/plugins/action/__init__.py
ansible_mitogen/plugins/action/mitogen_fetch.py
......@@ -37,6 +33,7 @@ ansible_mitogen/plugins/connection/mitogen_local.py
ansible_mitogen/plugins/connection/mitogen_lxc.py
ansible_mitogen/plugins/connection/mitogen_lxd.py
ansible_mitogen/plugins/connection/mitogen_machinectl.py
ansible_mitogen/plugins/connection/mitogen_podman.py
ansible_mitogen/plugins/connection/mitogen_setns.py
ansible_mitogen/plugins/connection/mitogen_ssh.py
ansible_mitogen/plugins/connection/mitogen_su.py
......@@ -46,6 +43,8 @@ ansible_mitogen/plugins/strategy/mitogen.py
ansible_mitogen/plugins/strategy/mitogen_free.py
ansible_mitogen/plugins/strategy/mitogen_host_pinned.py
ansible_mitogen/plugins/strategy/mitogen_linear.py
ansible_mitogen/utils/__init__.py
ansible_mitogen/utils/unsafe.py
mitogen/__init__.py
mitogen/buildah.py
mitogen/core.py
......@@ -62,6 +61,7 @@ mitogen/master.py
mitogen/minify.py
mitogen/os_fork.py
mitogen/parent.py
mitogen/podman.py
mitogen/profiler.py
mitogen/select.py
mitogen/service.py
......@@ -78,4 +78,5 @@ mitogen.egg-info/not-zip-safe
mitogen.egg-info/top_level.txt
mitogen/compat/__init__.py
mitogen/compat/pkgutil.py
mitogen/compat/tokenize.py
\ No newline at end of file
mitogen/compat/tokenize.py
tests/testlib.py
\ No newline at end of file
mitogen-0.2.9/mitogen/__init__.py → mitogen-0.3.7/mitogen/__init__.py
View file @ 985463fd
......@@ -35,7 +35,7 @@ be expected. On the slave, it is built dynamically during startup.
#: Library version as a tuple.
__version__ = (0, 2, 9)
__version__ = (0, 3, 7)
#: This is :data:`False` in slave contexts. Previously it was used to prevent
......
mitogen-0.2.9/mitogen/buildah.py → mitogen-0.3.7/mitogen/buildah.py
View file @ 985463fd
......@@ -30,7 +30,6 @@
import logging
import mitogen.core
import mitogen.parent
......
mitogen-0.2.9/mitogen/core.py → mitogen-0.3.7/mitogen/core.py
View file @ 985463fd
......@@ -34,6 +34,34 @@ non-essential code in order to reduce its size, since it is also serves as the
bootstrap implementation sent to every new slave context.
"""
import sys
try:
import _frozen_importlib_external
except ImportError:
pass
else:
class MonkeyPatchedPathFinder(_frozen_importlib_external.PathFinder):
"""
Meta path finder for sys.path and package __path__ attributes.
Patched for https://github.com/python/cpython/issues/115911.
"""
@classmethod
def _path_importer_cache(cls, path):
if path == '':
try:
path = _frozen_importlib_external._os.getcwd()
except (FileNotFoundError, PermissionError):
return None
return super()._path_importer_cache(path)
if sys.version_info[:2] <= (3, 12):
for i, mpf in enumerate(sys.meta_path):
if mpf is _frozen_importlib_external.PathFinder:
sys.meta_path[i] = MonkeyPatchedPathFinder
del i, mpf
import binascii
import collections
import encodings.latin_1
......@@ -49,18 +77,22 @@ import pstats
import signal
import socket
import struct
import sys
import syslog
import threading
import time
import traceback
import types
import warnings
import weakref
import zlib
# Python >3.7 deprecated the imp module.
warnings.filterwarnings('ignore', message='the imp module is deprecated')
import imp
try:
# Python >= 3.4, PEP 451 ModuleSpec API
import importlib.machinery
import importlib.util
except ImportError:
# Python < 3.4, PEP 302 Import Hooks
import imp
# Absolute imports for <2.5.
select = __import__('select')
......@@ -386,6 +418,20 @@ def _partition(s, sep, find):
return left, sep, s[len(left)+len(sep):]
def threading__current_thread():
try:
return threading.current_thread() # Added in Python 2.6+
except AttributeError:
return threading.currentThread() # Deprecated in Python 3.10+
def threading__thread_name(thread):
try:
return thread.name # Added in Python 2.6+
except AttributeError:
return thread.getName() # Deprecated in Python 3.10+
if hasattr(UnicodeType, 'rpartition'):
str_partition = UnicodeType.partition
str_rpartition = UnicodeType.rpartition
......@@ -1254,6 +1300,7 @@ class Importer(object):
'minify',
'os_fork',
'parent',
'podman',
'select',
'service',
'setns',
......@@ -1269,6 +1316,13 @@ class Importer(object):
# a negative round-trip.
'builtins',
'__builtin__',
# On some Python releases (e.g. 3.8, 3.9) the subprocess module tries
# to import of this Windows-only builtin module.
'msvcrt',
# Python 2.x module that was renamed to _thread in 3.x.
# This entry avoids a roundtrip on 2.x -> 3.x.
'thread',
# org.python.core imported by copy, pickle, xml.sax; breaks Jython, but
......@@ -1331,6 +1385,19 @@ class Importer(object):
def __repr__(self):
return 'Importer'
@staticmethod
def _loader_from_module(module, default=None):
"""Return the loader for a module object."""
try:
return module.__spec__.loader
except AttributeError:
pass
try:
return module.__loader__
except AttributeError:
pass
return default
def builtin_find_module(self, fullname):
# imp.find_module() will always succeed for __main__, because it is a
# built-in module. That means it exists on a special linked list deep
......@@ -1338,17 +1405,35 @@ class Importer(object):
if fullname == '__main__':
raise ModuleNotFoundError()
# For a module inside a package (e.g. pkg_a.mod_b) use the search path
# of that package (e.g. ['/usr/lib/python3.11/site-packages/pkg_a']).
parent, _, modname = str_rpartition(fullname, '.')
if parent:
path = sys.modules[parent].__path__
else:
path = None
# For a top-level module search builtin modules, frozen modules,
# system specific locations (e.g. Windows registry, site-packages).
# Otherwise use search path of the parent package.
# Works for both stdlib modules & third-party modules.
# If the search is unsuccessful then raises ImportError.
fp, pathname, description = imp.find_module(modname, path)
if fp:
fp.close()
def find_module(self, fullname, path=None):
"""
Return a loader (ourself) or None, for the module with fullname.
Implements importlib.abc.MetaPathFinder.find_module().
Deprecrated in Python 3.4+, replaced by find_spec().
Raises ImportWarning in Python 3.10+.
Removed in Python 3.12.
fullname Fully qualified module name, e.g. "os.path".
path __path__ of parent packge. None for a top level module.
"""
if hasattr(_tls, 'running'):
return None
......@@ -1356,14 +1441,13 @@ class Importer(object):
try:
#_v and self._log.debug('Python requested %r', fullname)
fullname = to_text(fullname)
pkgname, dot, _ = str_rpartition(fullname, '.')
pkgname, _, suffix = str_rpartition(fullname, '.')
pkg = sys.modules.get(pkgname)
if pkgname and getattr(pkg, '__loader__', None) is not self:
self._log.debug('%s is submodule of a locally loaded package',
fullname)
return None
suffix = fullname[len(pkgname+dot):]
if pkgname and suffix not in self._present.get(pkgname, ()):
self._log.debug('%s has no submodule %s', pkgname, suffix)
return None
......@@ -1383,6 +1467,66 @@ class Importer(object):
finally:
del _tls.running
def find_spec(self, fullname, path, target=None):
"""
Return a `ModuleSpec` for module with `fullname` if we will load it.
Otherwise return `None`, allowing other finders to try.
fullname Fully qualified name of the module (e.g. foo.bar.baz)
path Path entries to search. None for a top-level module.
target Existing module to be reloaded (if any).
Implements importlib.abc.MetaPathFinder.find_spec()
Python 3.4+.
"""
# Presence of _tls.running indicates we've re-invoked importlib.
# Abort early to prevent infinite recursion. See below.
if hasattr(_tls, 'running'):
return None
log = self._log.getChild('find_spec')
if fullname.endswith('.'):
return None
pkgname, _, modname = fullname.rpartition('.')
if pkgname and modname not in self._present.get(pkgname, ()):
log.debug('Skipping %s. Parent %s has no submodule %s',
fullname, pkgname, modname)
return None
pkg = sys.modules.get(pkgname)
pkg_loader = self._loader_from_module(pkg)
if pkgname and pkg_loader is not self:
log.debug('Skipping %s. Parent %s was loaded by %r',
fullname, pkgname, pkg_loader)
return None
# #114: whitelisted prefixes override any system-installed package.
if self.whitelist != ['']:
if any(s and fullname.startswith(s) for s in self.whitelist):
log.debug('Handling %s. It is whitelisted', fullname)
return importlib.machinery.ModuleSpec(fullname, loader=self)
if fullname == '__main__':
log.debug('Handling %s. A special case', fullname)
return importlib.machinery.ModuleSpec(fullname, loader=self)
# Re-invoke the import machinery to allow other finders to try.
# Set a guard, so we don't infinitely recurse. See top of this method.
_tls.running = True
try:
spec = importlib.util._find_spec(fullname, path, target)
finally:
del _tls.running
if spec:
log.debug('Skipping %s. Available as %r', fullname, spec)
return spec
log.debug('Handling %s. Unavailable locally', fullname)
return importlib.machinery.ModuleSpec(fullname, loader=self)
blacklisted_msg = (
'%r is present in the Mitogen importer blacklist, therefore this '
'context will not attempt to request it from the master, as the '
......@@ -1469,7 +1613,71 @@ class Importer(object):
if present:
callback()
def create_module(self, spec):
"""
Return a module object for the given ModuleSpec.
Implements PEP-451 importlib.abc.Loader API introduced in Python 3.4.
Unlike Loader.load_module() this shouldn't populate sys.modules or
set module attributes. Both are done by Python.
"""
self._log.debug('Creating module for %r', spec)
# FIXME Should this be done in find_spec()? Can it?
self._refuse_imports(spec.name)
# FIXME "create_module() should properly handle the case where it is
# called more than once for the same spec/module." -- PEP-451
event = threading.Event()
self._request_module(spec.name, callback=event.set)
event.wait()
# 0:fullname 1:pkg_present 2:path 3:compressed 4:related
_, pkg_present, path, _, _ = self._cache[spec.name]
if path is None:
raise ImportError(self.absent_msg % (spec.name))
spec.origin = self.get_filename(spec.name)
if pkg_present is not None:
# TODO Namespace packages
spec.submodule_search_locations = []
self._present[spec.name] = pkg_present
module = types.ModuleType(spec.name)
# FIXME create_module() shouldn't initialise module attributes
module.__file__ = spec.origin
return module
def exec_module(self, module):
"""
Execute the module to initialise it. Don't return anything.
Implements PEP-451 importlib.abc.Loader API, introduced in Python 3.4.
"""
name = module.__spec__.name
origin = module.__spec__.origin
self._log.debug('Executing %s from %s', name, origin)
source = self.get_source(name)
try:
# Compile the source into a code object. Don't add any __future__
# flags and don't inherit any from this module.
# FIXME Should probably be exposed as get_code()
code = compile(source, origin, 'exec', flags=0, dont_inherit=True)
except SyntaxError:
# FIXME Why is this LOG, rather than self._log?
LOG.exception('while importing %r', name)
raise
exec(code, module.__dict__)
def load_module(self, fullname):
"""
Return the loaded module specified by fullname.
Implements importlib.abc.Loader.load_module().
Deprecated in Python 3.4+, replaced by create_module() & exec_module().
"""
fullname = to_text(fullname)
_v and self._log.debug('requesting %s', fullname)
self._refuse_imports(fullname)
......@@ -1478,11 +1686,11 @@ class Importer(object):
self._request_module(fullname, event.set)
event.wait()
ret = self._cache[fullname]
if ret[2] is None:
# 0:fullname 1:pkg_present 2:path 3:compressed 4:related
_, pkg_present, path, _, _ = self._cache[fullname]
if path is None:
raise ModuleNotFoundError(self.absent_msg % (fullname,))
pkg_present = ret[1]
mod = sys.modules.setdefault(fullname, imp.new_module(fullname))
mod.__file__ = self.get_filename(fullname)
mod.__loader__ = self
......@@ -2679,7 +2887,7 @@ class Latch(object):
raise e
assert cookie == got_cookie, (
"Cookie incorrect; got %r, expected %r" \
"Cookie incorrect; got %r, expected %r"
% (binascii.hexlify(got_cookie),
binascii.hexlify(cookie))
)
......@@ -2734,7 +2942,7 @@ class Latch(object):
return 'Latch(%#x, size=%d, t=%r)' % (
id(self),
len(self._queue),
threading.currentThread().getName(),
threading__thread_name(threading__current_thread()),
)
......@@ -2801,7 +3009,7 @@ class Waker(Protocol):
self.stream.transmit_side.write(b(' '))
except OSError:
e = sys.exc_info()[1]
if e.args[0] in (errno.EBADF, errno.EWOULDBLOCK):
if e.args[0] not in (errno.EBADF, errno.EWOULDBLOCK):
raise
broker_shutdown_msg = (
......@@ -3634,7 +3842,6 @@ class Dispatcher(object):
self._service_recv.notify = None
self.recv.close()
@classmethod
@takes_econtext
def forget_chain(cls, chain_id, econtext):
......@@ -3860,7 +4067,7 @@ class ExternalContext(object):
else:
core_src_fd = self.config.get('core_src_fd', 101)
if core_src_fd:
fp = os.fdopen(core_src_fd, 'rb', 1)
fp = os.fdopen(core_src_fd, 'rb', 0)
try:
core_src = fp.read()
# Strip "ExternalContext.main()" call from last line.
......@@ -3884,7 +4091,7 @@ class ExternalContext(object):
def _setup_package(self):
global mitogen
mitogen = imp.new_module('mitogen')
mitogen = types.ModuleType('mitogen')
mitogen.__package__ = 'mitogen'
mitogen.__path__ = []
mitogen.__loader__ = self.importer
......
mitogen-0.2.9/mitogen/fakessh.py → mitogen-0.3.7/mitogen/fakessh.py
View file @ 985463fd
......@@ -103,7 +103,6 @@ import tempfile
import threading
import mitogen.core
import mitogen.master
import mitogen.parent
from mitogen.core import LOG, IOLOG
......@@ -200,7 +199,7 @@ class Process(object):
def _on_stdin(self, msg):
if msg.is_dead:
IOLOG.debug('%r._on_stdin() -> %r', self, data)
IOLOG.debug('%r._on_stdin() -> %r', self, msg)
self.pump.protocol.close()
return
......@@ -437,7 +436,7 @@ def run(dest, router, args, deadline=None, econtext=None):
fp.write(inspect.getsource(mitogen.core))
fp.write('\n')
fp.write('ExternalContext(%r).main()\n' % (
_get_econtext_config(context, sock2),
_get_econtext_config(econtext, sock2),
))
finally:
fp.close()
......
mitogen-0.2.9/mitogen/kubectl.py → mitogen-0.3.7/mitogen/kubectl.py
View file @ 985463fd
......@@ -28,7 +28,6 @@
# !mitogen: minify_safe
import mitogen.core
import mitogen.parent
......
mitogen-0.2.9/mitogen/lxc.py → mitogen-0.3.7/mitogen/lxc.py
View file @ 985463fd
......@@ -28,7 +28,6 @@
# !mitogen: minify_safe
import mitogen.core
import mitogen.parent
......
mitogen-0.2.9/mitogen/lxd.py → mitogen-0.3.7/mitogen/lxd.py
View file @ 985463fd
......@@ -28,7 +28,6 @@
# !mitogen: minify_safe
import mitogen.core
import mitogen.parent
......
mitogen-0.2.9/mitogen/master.py → mitogen-0.3.7/mitogen/master.py
View file @ 985463fd
......@@ -37,7 +37,6 @@ contexts.
import dis
import errno
import imp
import inspect
import itertools
import logging
......@@ -50,6 +49,16 @@ import threading
import types
import zlib
try:
# Python >= 3.4, PEP 451 ModuleSpec API
import importlib.machinery
import importlib.util
from _imp import is_builtin as _is_builtin
except ImportError:
# Python < 3.4, PEP 302 Import Hooks
import imp
from imp import is_builtin as _is_builtin
try:
import sysconfig
except ImportError:
......@@ -89,6 +98,14 @@ except NameError:
RLOG = logging.getLogger('mitogen.ctx')
# there are some cases where modules are loaded in memory only, such as
# ansible collections, and the module "filename" doesn't actually exist
SPECIAL_FILE_PATHS = {
"__synthetic__",
"<ansible_synthetic_collection_package>"
}
def _stdlib_paths():
"""
Return a set of paths from which Python imports the standard library.
......@@ -100,7 +117,7 @@ def _stdlib_paths():
]
prefixes = (getattr(sys, a, None) for a in attr_candidates)
version = 'python%s.%s' % sys.version_info[0:2]
s = set(os.path.abspath(os.path.join(p, 'lib', version))
s = set(os.path.realpath(os.path.join(p, 'lib', version))
for p in prefixes if p is not None)
# When running 'unit2 tests/module_finder_test.py' in a Py2 venv on Ubuntu
......@@ -114,7 +131,16 @@ def is_stdlib_name(modname):
"""
Return :data:`True` if `modname` appears to come from the standard library.
"""
if imp.is_builtin(modname) != 0:
# `(_imp|imp).is_builtin()` isn't a documented part of Python's stdlib.
# Returns 1 if modname names a module that is "builtin" to the the Python
# interpreter (e.g. '_sre'). Otherwise 0 (e.g. 're', 'netifaces').
#
# """
# Main is a little special - imp.is_builtin("__main__") will return False,
# but BuiltinImporter is still the most appropriate initial setting for
# its __loader__ attribute.
# """ -- comment in CPython pylifecycle.c:add_main_module()
if _is_builtin(modname) != 0:
return True
module = sys.modules.get(modname)
......@@ -138,7 +164,7 @@ def is_stdlib_path(path):
)
def get_child_modules(path):
def get_child_modules(path, fullname):
"""
Return the suffixes of submodules directly neated beneath of the package
directory at `path`.
......@@ -147,12 +173,19 @@ def get_child_modules(path):
Path to the module's source code on disk, or some PEP-302-recognized
equivalent. Usually this is the module's ``__file__`` attribute, but
is specified explicitly to avoid loading the module.
:param str fullname:
Name of the package we're trying to get child modules for
:return:
List of submodule name suffixes.
"""
it = pkgutil.iter_modules([os.path.dirname(path)])
return [to_text(name) for _, name, _ in it]
mod_path = os.path.dirname(path)
if mod_path != '':
return [to_text(name) for _, name, _ in pkgutil.iter_modules([mod_path])]
else:
# we loaded some weird package in memory, so we'll see if it has a custom loader we can use
loader = pkgutil.find_loader(fullname)
return [to_text(name) for name, _ in loader.iter_modules(None)] if loader else []
def _looks_like_script(path):
......@@ -177,17 +210,31 @@ def _looks_like_script(path):
def _py_filename(path):
"""
Returns a tuple of a Python path (if the file looks Pythonic) and whether or not
the Python path is special. Special file paths/modules might only exist in memory
"""
if not path:
return None
return None, False
if path[-4:] in ('.pyc', '.pyo'):
path = path.rstrip('co')
if path.endswith('.py'):
return path
return path, False
if os.path.exists(path) and _looks_like_script(path):
return path
return path, False
basepath = os.path.basename(path)
if basepath in SPECIAL_FILE_PATHS:
return path, True
# return None, False means that the filename passed to _py_filename does not appear
# to be python, and code later will handle when this function returns None
# see https://github.com/dw/mitogen/pull/715#discussion_r532380528 for how this
# decision was made to handle non-python files in this manner
return None, False
def _get_core_source():
......@@ -424,6 +471,9 @@ class FinderMethod(object):
name according to the running Python interpreter. You'd think this was a
simple task, right? Naive young fellow, welcome to the real world.
"""
def __init__(self):
self.log = LOG.getChild(self.__class__.__name__)
def __repr__(self):
return '%s()' % (type(self).__name__,)
......@@ -483,38 +533,57 @@ class PkgutilMethod(FinderMethod):
Find `fullname` using :func:`pkgutil.find_loader`.
"""
try:
# If fullname refers to a submodule that's not already imported
# then the containing package is imported.
# Pre-'import spec' this returned None, in Python3.6 it raises
# ImportError.
loader = pkgutil.find_loader(fullname)
except ImportError:
e = sys.exc_info()[1]
LOG.debug('%r._get_module_via_pkgutil(%r): %s',
self, fullname, e)
LOG.debug('%r: find_loader(%r) failed: %s', self, fullname, e)
return None
IOLOG.debug('%r._get_module_via_pkgutil(%r) -> %r',
self, fullname, loader)
if not loader:
LOG.debug('%r: find_loader(%r) returned %r, aborting',
self, fullname, loader)
return
try:
path = _py_filename(loader.get_filename(fullname))
source = loader.get_source(fullname)
is_pkg = loader.is_package(fullname)
except (AttributeError, ImportError):
# - Per PEP-302, get_source() and is_package() are optional,
# calling them may throw AttributeError.
path = loader.get_filename(fullname)
except (AttributeError, ImportError, ValueError):
# - get_filename() may throw ImportError if pkgutil.find_loader()
# picks a "parent" package's loader for some crap that's been
# stuffed in sys.modules, for example in the case of urllib3:
# "loader for urllib3.contrib.pyopenssl cannot handle
# requests.packages.urllib3.contrib.pyopenssl"
e = sys.exc_info()[1]
LOG.debug('%r: loading %r using %r failed: %s',
self, fullname, loader, e)
LOG.debug('%r: %r.get_file_name(%r) failed: %r', self, loader, fullname, e)
return
path, is_special = _py_filename(path)
try:
source = loader.get_source(fullname)
except AttributeError:
# Per PEP-302, get_source() is optional,
e = sys.exc_info()[1]
LOG.debug('%r: %r.get_source() failed: %r', self, loader, fullname, e)
return
try:
is_pkg = loader.is_package(fullname)
except AttributeError:
# Per PEP-302, is_package() is optional,
e = sys.exc_info()[1]
LOG.debug('%r: %r.is_package(%r) failed: %r', self, loader, fullname, e)
return
# workaround for special python modules that might only exist in memory
if is_special and is_pkg and not source:
source = '\n'
if path is None or source is None:
LOG.debug('%r: path=%r, source=%r, aborting', self, path, source)
return
if isinstance(source, mitogen.core.UnicodeType):
......@@ -534,23 +603,37 @@ class SysModulesMethod(FinderMethod):
"""
Find `fullname` using its :data:`__file__` attribute.
"""
module = sys.modules.get(fullname)
try:
module = sys.modules[fullname]
except KeyError:
LOG.debug('%r: sys.modules[%r] absent, aborting', self, fullname)
return
if not isinstance(module, types.ModuleType):
LOG.debug('%r: sys.modules[%r] absent or not a regular module',
self, fullname)
LOG.debug('%r: sys.modules[%r] is %r, aborting',
self, fullname, module)
return
try:
resolved_name = module.__name__
except AttributeError:
LOG.debug('%r: %r has no __name__, aborting', self, module)
return
LOG.debug('_get_module_via_sys_modules(%r) -> %r', fullname, module)
alleged_name = getattr(module, '__name__', None)
if alleged_name != fullname:
LOG.debug('sys.modules[%r].__name__ is incorrect, assuming '
'this is a hacky module alias and ignoring it. '
'Got %r, module object: %r',
fullname, alleged_name, module)
if resolved_name != fullname:
LOG.debug('%r: %r.__name__ is %r, aborting',
self, module, resolved_name)
return
path = _py_filename(getattr(module, '__file__', ''))
try:
path = module.__file__
except AttributeError:
LOG.debug('%r: %r has no __file__, aborting', self, module)
return
path, _ = _py_filename(path)
if not path:
LOG.debug('%r: %r.__file__ is %r, aborting', self, module, path)
return
LOG.debug('%r: sys.modules[%r]: found %s', self, fullname, path)
......@@ -572,7 +655,7 @@ class SysModulesMethod(FinderMethod):
return path, source, is_pkg
class ParentEnumerationMethod(FinderMethod):
class ParentImpEnumerationMethod(FinderMethod):
"""
Attempt to fetch source code by examining the module's (hopefully less
insane) parent package, and if no insane parents exist, simply use
......@@ -595,10 +678,24 @@ class ParentEnumerationMethod(FinderMethod):
module object or any parent package's :data:`__path__`, since they have all
been overwritten. Some men just want to watch the world burn.
"""
@staticmethod
def _iter_parents(fullname):
"""
>>> list(ParentEnumerationMethod._iter_parents('a'))
[('', 'a')]
>>> list(ParentEnumerationMethod._iter_parents('a.b.c'))
[('a.b', 'c'), ('a', 'b'), ('', 'a')]
"""
while fullname:
fullname, _, modname = str_rpartition(fullname, u'.')
yield fullname, modname
def _find_sane_parent(self, fullname):
"""
Iteratively search :data:`sys.modules` for the least indirect parent of
`fullname` that is loaded and contains a :data:`__path__` attribute.
`fullname` that's from the same package and has a :data:`__path__`
attribute.
:return:
`(parent_name, path, modpath)` tuple, where:
......@@ -611,21 +708,40 @@ class ParentEnumerationMethod(FinderMethod):
* `modpath`: list of module name components leading from `path`
to the target module.
"""
path = None
modpath = []
while True:
pkgname, _, modname = str_rpartition(to_text(fullname), u'.')
for pkgname, modname in self._iter_parents(fullname):
modpath.insert(0, modname)
if not pkgname:
return [], None, modpath
pkg = sys.modules.get(pkgname)
path = getattr(pkg, '__path__', None)
if pkg and path:
return pkgname.split('.'), path, modpath
try:
pkg = sys.modules[pkgname]
except KeyError:
LOG.debug('%r: sys.modules[%r] absent, skipping', self, pkgname)
continue
try:
resolved_pkgname = pkg.__name__
except AttributeError:
LOG.debug('%r: %r has no __name__, skipping', self, pkg)
continue
if resolved_pkgname != pkgname:
LOG.debug('%r: %r.__name__ is %r, skipping',
self, pkg, resolved_pkgname)
continue
try:
path = pkg.__path__
except AttributeError:
LOG.debug('%r: %r has no __path__, skipping', self, pkg)
continue
if not path:
LOG.debug('%r: %r.__path__ is %r, skipping', self, pkg, path)
continue
LOG.debug('%r: %r lacks __path__ attribute', self, pkgname)
fullname = pkgname
return pkgname.split('.'), path, modpath
def _found_package(self, fullname, path):
path = os.path.join(path, '__init__.py')
......@@ -639,7 +755,7 @@ class ParentEnumerationMethod(FinderMethod):
def _found_module(self, fullname, path, fp, is_pkg=False):
try:
path = _py_filename(path)
path, _ = _py_filename(path)
if not path:
return
......@@ -657,6 +773,7 @@ class ParentEnumerationMethod(FinderMethod):
def _find_one_component(self, modname, search_path):
try:
#fp, path, (suffix, _, kind) = imp.find_module(modname, search_path)
# FIXME The imp module was removed in Python 3.12.
return imp.find_module(modname, search_path)
except ImportError:
e = sys.exc_info()[1]
......@@ -668,6 +785,9 @@ class ParentEnumerationMethod(FinderMethod):
"""
See implementation for a description of how this works.
"""
if sys.version_info >= (3, 4):
return None
#if fullname not in sys.modules:
# Don't attempt this unless a module really exists in sys.modules,
# else we could return junk.
......@@ -696,6 +816,99 @@ class ParentEnumerationMethod(FinderMethod):
return self._found_module(fullname, path, fp)
class ParentSpecEnumerationMethod(ParentImpEnumerationMethod):
def _find_parent_spec(self, fullname):
#history = []
debug = self.log.debug
children = []
for parent_name, child_name in self._iter_parents(fullname):
children.insert(0, child_name)
if not parent_name:
debug('abandoning %r, reached top-level', fullname)
return None, children
try:
parent = sys.modules[parent_name]
except KeyError:
debug('skipping %r, not in sys.modules', parent_name)
continue
try:
spec = parent.__spec__
except AttributeError:
debug('skipping %r: %r.__spec__ is absent',
parent_name, parent)
continue
if not spec:
debug('skipping %r: %r.__spec__=%r',
parent_name, parent, spec)
continue
if spec.name != parent_name:
debug('skipping %r: %r.__spec__.name=%r does not match',
parent_name, parent, spec.name)
continue
if not spec.submodule_search_locations:
debug('skipping %r: %r.__spec__.submodule_search_locations=%r',
parent_name, parent, spec.submodule_search_locations)
continue
return spec, children
raise ValueError('%s._find_parent_spec(%r) unexpectedly reached bottom'
% (self.__class__.__name__, fullname))
def find(self, fullname):
# Returns absolute path, ParentImpEnumerationMethod returns relative
# >>> spec_pem.find('six_brokenpkg._six')[::2]
# ('/Users/alex/src/mitogen/tests/data/importer/six_brokenpkg/_six.py', False)
if sys.version_info < (3, 4):
return None
fullname = to_text(fullname)
spec, children = self._find_parent_spec(fullname)
for child_name in children:
if spec:
name = '%s.%s' % (spec.name, child_name)
submodule_search_locations = spec.submodule_search_locations
else:
name = child_name
submodule_search_locations = None
spec = importlib.util._find_spec(name, submodule_search_locations)
if spec is None:
self.log.debug('%r spec unavailable from %s', fullname, spec)
return None
is_package = spec.submodule_search_locations is not None
if name != fullname:
if not is_package:
self.log.debug('%r appears to be child of non-package %r',
fullname, spec)
return None
continue
if not spec.has_location:
self.log.debug('%r.origin cannot be read as a file', spec)
return None
if os.path.splitext(spec.origin)[1] != '.py':
self.log.debug('%r.origin does not contain Python source code',
spec)
return None
# FIXME This should use loader.get_source()
with open(spec.origin, 'rb') as f:
source = f.read()
return spec.origin, source, is_package
raise ValueError('%s.find(%r) unexpectedly reached bottom'
% (self.__class__.__name__, fullname))
class ModuleFinder(object):
"""
Given the name of a loaded module, make a best-effort attempt at finding
......@@ -736,7 +949,8 @@ class ModuleFinder(object):
DefectivePython3xMainMethod(),
PkgutilMethod(),
SysModulesMethod(),
ParentEnumerationMethod(),
ParentSpecEnumerationMethod(),
ParentImpEnumerationMethod(),
]
def get_module_source(self, fullname):
......@@ -971,7 +1185,7 @@ class ModuleResponder(object):
self.minify_secs += mitogen.core.now() - t0
if is_pkg:
pkg_present = get_child_modules(path)
pkg_present = get_child_modules(path, fullname)
self._log.debug('%s is a package at %s with submodules %r',
fullname, path, pkg_present)
else:
......@@ -1134,7 +1348,7 @@ class Broker(mitogen.core.Broker):
def __init__(self, install_watcher=True):
if install_watcher:
self._watcher = ThreadWatcher.watch(
target=threading.currentThread(),
target=mitogen.core.threading__current_thread(),
on_join=self.shutdown,
)
super(Broker, self).__init__()
......@@ -1279,7 +1493,8 @@ class Router(mitogen.parent.Router):
self.broker.defer(stream.on_disconnect, self.broker)
def disconnect_all(self):
for stream in self._stream_by_id.values():
# making stream_by_id python3-safe by converting stream_by_id values iter to list
for stream in list(self._stream_by_id.values()):
self.disconnect_stream(stream)
......
mitogen-0.2.9/mitogen/os_fork.py → mitogen-0.3.7/mitogen/os_fork.py
View file @ 985463fd
......@@ -35,7 +35,6 @@ Support for operating in a mixed threading/forking environment.
import os
import socket
import sys
import threading
import weakref
import mitogen.core
......@@ -158,7 +157,7 @@ class Corker(object):
held. This will not return until each thread acknowledges it has ceased
execution.
"""
current = threading.currentThread()
current = mitogen.core.threading__current_thread()
s = mitogen.core.b('CORK') * ((128 // 4) * 1024)
self._rsocks = []
......
mitogen-0.2.9/mitogen/parent.py → mitogen-0.3.7/mitogen/parent.py
View file @ 985463fd
......@@ -34,7 +34,7 @@ sent to any child context that is due to become a parent, due to recursive
connection.
"""
import codecs
import binascii
import errno
import fcntl
import getpass
......@@ -1027,7 +1027,7 @@ class KqueuePoller(mitogen.core.Poller):
class EpollPoller(mitogen.core.Poller):
"""
Poller based on the Linux :linux:man2:`epoll` interface.
Poller based on the Linux :linux:man7:`epoll` interface.
"""
SUPPORTED = hasattr(select, 'epoll')
_repr = 'EpollPoller()'
......@@ -1405,13 +1405,24 @@ class Connection(object):
# file descriptor 0 as 100, creates a pipe, then execs a new interpreter
# with a custom argv.
# * Optimized for minimum byte count after minification & compression.
# The script preamble_size.py measures this.
# * 'CONTEXT_NAME' and 'PREAMBLE_COMPRESSED_LEN' are substituted with
# their respective values.
# * CONTEXT_NAME must be prefixed with the name of the Python binary in
# order to allow virtualenvs to detect their install prefix.
# * For Darwin, OS X installs a craptacular argv0-introspecting Python
# version switcher as /usr/bin/python. Override attempts to call it
# with an explicit call to python2.7
#
# macOS tweaks for Python 2.7 must be kept in sync with the the Ansible
# module test_echo_module, used by the integration tests.
# * macOS <= 10.14 (Darwin <= 18) install an unreliable Python version
# switcher as /usr/bin/python, which introspects argv0. To workaround
# it we redirect attempts to call /usr/bin/python with an explicit
# call to /usr/bin/python2.7. macOS 10.15 (Darwin 19) removed it.
# * macOS 11.x (Darwin 20, Big Sur) and macOS 12.x (Darwin 21, Montery)
# do something slightly different. The Python executable is patched to
# perform an extra execvp(). I don't fully understand the details, but
# setting PYTHON_LAUNCHED_FROM_WRAPPER=1 avoids it.
# * macOS 12.3+ (Darwin 21.4+, Monterey) doesn't ship Python.
# https://developer.apple.com/documentation/macos-release-notes/macos-12_3-release-notes#Python
#
# Locals:
# R: read side of interpreter stdin.
......@@ -1434,12 +1445,12 @@ class Connection(object):
os.close(r)
os.close(W)
os.close(w)
if sys.platform == 'darwin' and sys.executable == '/usr/bin/python':
sys.executable += sys.version[:3]
if os.uname()[0]=='Darwin'and os.uname()[2][:2]<'19'and sys.executable=='/usr/bin/python':sys.executable='/usr/bin/python2.7'
if os.uname()[0]=='Darwin'and os.uname()[2][:2]in'2021'and sys.version[:3]=='2.7':os.environ['PYTHON_LAUNCHED_FROM_WRAPPER']='1'
os.environ['ARGV0']=sys.executable
os.execl(sys.executable,sys.executable+'(mitogen:CONTEXT_NAME)')
os.write(1,'MITO000\n'.encode())
C=_(os.fdopen(0,'rb').read(PREAMBLE_COMPRESSED_LEN),'zip')
C=zlib.decompress(os.fdopen(0,'rb').read(PREAMBLE_COMPRESSED_LEN))
fp=os.fdopen(W,'wb',0)
fp.write(C)
fp.close()
......@@ -1465,22 +1476,22 @@ class Connection(object):
def get_boot_command(self):
source = inspect.getsource(self._first_stage)
source = textwrap.dedent('\n'.join(source.strip().split('\n')[2:]))
source = source.replace(' ', '\t')
source = source.replace(' ', ' ')
source = source.replace('CONTEXT_NAME', self.options.remote_name)
preamble_compressed = self.get_preamble()
source = source.replace('PREAMBLE_COMPRESSED_LEN',
str(len(preamble_compressed)))
compressed = zlib.compress(source.encode(), 9)
encoded = codecs.encode(compressed, 'base64').replace(b('\n'), b(''))
# We can't use bytes.decode() in 3.x since it was restricted to always
# return unicode, so codecs.decode() is used instead. In 3.x
# codecs.decode() requires a bytes object. Since we must be compatible
# with 2.4 (no bytes literal), an extra .encode() either returns the
# same str (2.x) or an equivalent bytes (3.x).
encoded = binascii.b2a_base64(compressed).replace(b('\n'), b(''))
# Just enough to decode, decompress, and exec the first stage.
# Priorities: wider compatibility, faster startup, shorter length.
# `import os` here, instead of stage 1, to save a few bytes.
# `sys.path=...` for https://github.com/python/cpython/issues/115911.
return self.get_python_argv() + [
'-c',
'import codecs,os,sys;_=codecs.decode;'
'exec(_(_("%s".encode(),"base64"),"zip"))' % (encoded.decode(),)
'import sys;sys.path=[p for p in sys.path if p];import binascii,os,zlib;'
'exec(zlib.decompress(binascii.a2b_base64("%s")))' % (encoded.decode(),),
]
def get_econtext_config(self):
......@@ -1502,7 +1513,7 @@ class Connection(object):
def get_preamble(self):
suffix = (
'\nExternalContext(%r).main()\n' %\
'\nExternalContext(%r).main()\n' %
(self.get_econtext_config(),)
)
partial = get_core_source_partial()
......@@ -2501,6 +2512,9 @@ class Router(mitogen.core.Router):
def ssh(self, **kwargs):
return self.connect(u'ssh', **kwargs)
def podman(self, **kwargs):
return self.connect(u'podman', **kwargs)
class Reaper(object):
"""
......
mitogen-0.3.7/mitogen/podman.py 0 → 100644
View file @ 985463fd
# Copyright 2019, David Wilson
# Copyright 2021, Mitogen contributors
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
# this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright notice,
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# 3. Neither the name of the copyright holder nor the names of its contributors
# may be used to endorse or promote products derived from this software without
# specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
# !mitogen: minify_safe
import logging
import mitogen.parent
LOG = logging.getLogger(__name__)
class Options(mitogen.parent.Options):
container = None
username = None
podman_path = 'podman'
def __init__(self, container=None, podman_path=None, username=None,
**kwargs):
super(Options, self).__init__(**kwargs)
assert container is not None
self.container = container
if podman_path:
self.podman_path = podman_path
if username:
self.username = username
class Connection(mitogen.parent.Connection):
options_class = Options
child_is_immediate_subprocess = False
# TODO: better way of capturing errors such as "No such container."
create_child_args = {
'merge_stdio': True
}
def _get_name(self):
return u'podman.' + self.options.container
def get_boot_command(self):
args = [self.options.podman_path, 'exec']
if self.options.username:
args += ['--user=' + self.options.username]
args += ["--interactive", "--", self.options.container]
return args + super(Connection, self).get_boot_command()
mitogen-0.2.9/mitogen/profiler.py → mitogen-0.3.7/mitogen/profiler.py
View file @ 985463fd
......@@ -90,7 +90,7 @@ def merge_stats(outpath, inpaths):
break
time.sleep(0.2)
stats.dump_stats(outpath)
pstats.dump_stats(outpath)
def generate_stats(outpath, tmpdir):
......
mitogen-0.2.9/mitogen/service.py → mitogen-0.3.7/mitogen/service.py
View file @ 985463fd
......@@ -31,7 +31,6 @@
import grp
import logging
import os
import os.path
import pprint
import pwd
import stat
......@@ -74,7 +73,7 @@ else:
@mitogen.core.takes_router
def get_or_create_pool(size=None, router=None):
def get_or_create_pool(size=None, router=None, context=None):
global _pool
global _pool_pid
......@@ -84,6 +83,12 @@ def get_or_create_pool(size=None, router=None):
_pool_lock.acquire()
try:
if _pool_pid != my_pid:
if router is None:
# fallback to trying to get router from context if that exists
if context is not None:
router = context.router
else:
raise ValueError("Unable to create Pool! Missing router.")
_pool = Pool(
router,
services=[],
......@@ -103,7 +108,8 @@ def get_or_create_pool(size=None, router=None):
def get_thread_name():
return threading.currentThread().getName()
thread = mitogen.core.threading__current_thread()
return mitogen.core.threading__thread_name(thread)
def call(service_name, method_name, call_context=None, **kwargs):
......@@ -119,7 +125,7 @@ def call(service_name, method_name, call_context=None, **kwargs):
if call_context:
return call_context.call_service(service_name, method_name, **kwargs)
else:
pool = get_or_create_pool()
pool = get_or_create_pool(context=kwargs.get('context'))
invoker = pool.get_invoker(service_name, msg=None)
return getattr(invoker.service, method_name)(**kwargs)
......@@ -685,6 +691,7 @@ class PushFileService(Service):
super(PushFileService, self).__init__(**kwargs)
self._lock = threading.Lock()
self._cache = {}
self._extra_sys_paths = set()
self._waiters = {}
self._sent_by_stream = {}
......@@ -738,30 +745,59 @@ class PushFileService(Service):
@arg_spec({
'context': mitogen.core.Context,
'paths': list,
'modules': list,
# 'modules': list, TODO, modules was passed into this func but it's not used yet
})
def propagate_paths_and_modules(self, context, paths, modules):
def propagate_paths_and_modules(self, context, paths, overridden_sources=None, extra_sys_paths=None):
"""
One size fits all method to ensure a target context has been preloaded
with a set of small files and Python modules.
:param dict overridden_sources:
Optional dict containing source code to override path's source code
:param extra_sys_paths:
Loads additional sys paths for use in finding modules; beneficial
in situations like loading Ansible Collections because source code
dependencies come from different file paths than where the source lives
"""
for path in paths:
self.propagate_to(context, mitogen.core.to_text(path))
#self.router.responder.forward_modules(context, modules) TODO
overridden_source = None
if overridden_sources is not None and path in overridden_sources:
overridden_source = overridden_sources[path]
self.propagate_to(context, mitogen.core.to_text(path), overridden_source)
# self.router.responder.forward_modules(context, modules) TODO
# NOTE: could possibly be handled by the above TODO, but not sure how forward_modules works enough
# to know for sure, so for now going to pass the sys paths themselves and have `propagate_to`
# load them up in sys.path for later import
# ensure we don't add to sys.path the same path we've already seen
for extra_path in extra_sys_paths:
# store extra paths in cached set for O(1) lookup
if extra_path not in self._extra_sys_paths:
# not sure if it matters but we could prepend to sys.path instead if we need to
sys.path.append(extra_path)
self._extra_sys_paths.add(extra_path)
@expose(policy=AllowParents())
@arg_spec({
'context': mitogen.core.Context,
'path': mitogen.core.FsPathTypes,
})
def propagate_to(self, context, path):
def propagate_to(self, context, path, overridden_source=None):
"""
If the optional parameter 'overridden_source' is passed, use
that instead of the path's code as source code. This works around some bugs
of source modules such as relative imports on unsupported Python versions
"""
if path not in self._cache:
LOG.debug('caching small file %s', path)
fp = open(path, 'rb')
try:
self._cache[path] = mitogen.core.Blob(fp.read())
finally:
fp.close()
if overridden_source is None:
fp = open(path, 'rb')
try:
self._cache[path] = mitogen.core.Blob(fp.read())
finally:
fp.close()
else:
self._cache[path] = mitogen.core.Blob(overridden_source)
self._forward(context, path)
@expose(policy=AllowParents())
......
mitogen-0.2.9/mitogen/ssh.py → mitogen-0.3.7/mitogen/ssh.py
View file @ 985463fd
......@@ -72,7 +72,10 @@ PASSWORD_PROMPT_PATTERN = re.compile(
)
HOSTKEY_REQ_PATTERN = re.compile(
b(r'are you sure you want to continue connecting \(yes/no\)\?'),
b(
r'are you sure you want to continue connecting '
r'\(yes/no(?:/\[fingerprint\])?\)\?'
),
re.I
)
......@@ -221,6 +224,14 @@ class Connection(mitogen.parent.Connection):
child_is_immediate_subprocess = False
# strings that, if escaped, cause problems creating connections
# example: `source /opt/rh/rh-python36/enable && python`
# is an acceptable ansible_python_version but shlex would quote the &&
# and prevent python from executing
SHLEX_IGNORE = [
"&&"
]
def _get_name(self):
s = u'ssh.' + mitogen.core.to_text(self.options.hostname)
if self.options.port and self.options.port != 22:
......@@ -291,4 +302,9 @@ class Connection(mitogen.parent.Connection):
bits += self.options.ssh_args
bits.append(self.options.hostname)
base = super(Connection, self).get_boot_command()
return bits + [shlex_quote(s).strip() for s in base]
base_parts = []
for s in base:
val = s if s in self.SHLEX_IGNORE else shlex_quote(s).strip()
base_parts.append(val)
return bits + base_parts
mitogen-0.2.9/mitogen/sudo.py → mitogen-0.3.7/mitogen/sudo.py
View file @ 985463fd
......@@ -256,6 +256,8 @@ class Connection(mitogen.parent.Connection):
# Note: sudo did not introduce long-format option processing until July
# 2013, so even though we parse long-format options, supply short-form
# to the sudo command.
boot_cmd = super(Connection, self).get_boot_command()
bits = [self.options.sudo_path, '-u', self.options.username]
if self.options.preserve_env:
bits += ['-E']
......@@ -268,4 +270,25 @@ class Connection(mitogen.parent.Connection):
if self.options.selinux_type:
bits += ['-t', self.options.selinux_type]
return bits + ['--'] + super(Connection, self).get_boot_command()
# special handling for bash builtins
# TODO: more efficient way of doing this, at least
# it's only 1 iteration of boot_cmd to go through
source_found = False
for cmd in boot_cmd[:]:
# rip `source` from boot_cmd if it exists; sudo.py can't run this
# even with -i or -s options
# since we've already got our ssh command working we shouldn't
# need to source anymore
# couldn't figure out how to get this to work using sudo flags
if 'source' == cmd:
boot_cmd.remove(cmd)
source_found = True
continue
if source_found:
# remove words until we hit the python interpreter call
if not cmd.endswith('python'):
boot_cmd.remove(cmd)
else:
break
return bits + ['--'] + boot_cmd
mitogen-0.2.9/mitogen/utils.py → mitogen-0.3.7/mitogen/utils.py
View file @ 985463fd
......@@ -29,14 +29,13 @@
# !mitogen: minify_safe
import datetime
import functools
import logging
import os
import sys
import mitogen
import mitogen.core
import mitogen.master
import mitogen.parent
iteritems = getattr(dict, 'iteritems', dict.items)
......@@ -173,12 +172,9 @@ def with_router(func):
do_stuff(blah, 123)
"""
@functools.wraps(func)
def wrapper(*args, **kwargs):
return run_with_router(func, *args, **kwargs)
if mitogen.core.PY3:
wrapper.func_name = func.__name__
else:
wrapper.func_name = func.func_name
return wrapper
......@@ -194,10 +190,13 @@ PASSTHROUGH = (
def cast(obj):
"""
Return obj (or a copy) with subtypes of builtins cast to their supertype.
Subtypes of those in :data:`PASSTHROUGH` are not modified.
Many tools love to subclass built-in types in order to implement useful
functionality, such as annotating the safety of a Unicode string, or adding
additional methods to a dict. However, cPickle loves to preserve those
subtypes during serialization, resulting in CallError during :meth:`call
additional methods to a dict. However :py:mod:`pickle` serializes these
exactly, leading to :exc:`mitogen.CallError` during :meth:`Context.call
<mitogen.parent.Context.call>` in the target when it tries to deserialize
the data.
......@@ -205,6 +204,9 @@ def cast(obj):
custom sub-types removed. The functionality is not default since the
resulting walk may be computationally expensive given a large enough graph.
Raises :py:exc:`TypeError` if an unknown subtype is encountered, or
casting does not return the desired supertype.
See :ref:`serialization-rules` for a list of supported types.
:param obj:
......@@ -219,8 +221,16 @@ def cast(obj):
if isinstance(obj, PASSTHROUGH):
return obj
if isinstance(obj, mitogen.core.UnicodeType):
return mitogen.core.UnicodeType(obj)
return _cast(obj, mitogen.core.UnicodeType)
if isinstance(obj, mitogen.core.BytesType):
return mitogen.core.BytesType(obj)
return _cast(obj, mitogen.core.BytesType)
raise TypeError("Cannot serialize: %r: %r" % (type(obj), obj))
def _cast(obj, desired_type):
result = desired_type(obj)
if type(result) is not desired_type:
raise TypeError("Cast of %r to %r failed, got %r"
% (type(obj), desired_type, type(result)))
return result
mitogen-0.2.9/setup.cfg → mitogen-0.3.7/setup.cfg
View file @ 985463fd
[bdist_wheel]
universal = 1
[coverage:run]
branch = true
source =
......
mitogen-0.2.9/setup.py → mitogen-0.3.7/setup.py
View file @ 985463fd
......@@ -26,6 +26,7 @@
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
import ast
import os
from setuptools import find_packages, setup
......@@ -37,29 +38,47 @@ def grep_version():
for line in fp:
if line.startswith('__version__'):
_, _, s = line.partition('=')
return '.'.join(map(str, eval(s)))
parts = ast.literal_eval(s.strip())
return '.'.join(str(part) for part in parts)
def long_description():
here = os.path.dirname(__file__)
readme_path = os.path.join(here, 'README.md')
with open(readme_path) as fp:
readme = fp.read()
return readme
setup(
name = 'mitogen',
version = grep_version(),
description = 'Library for writing distributed self-replicating programs.',
long_description = long_description(),
long_description_content_type='text/markdown',
author = 'David Wilson',
license = 'New BSD',
url = 'https://github.com/dw/mitogen/',
url = 'https://github.com/mitogen-hq/mitogen/',
packages = find_packages(exclude=['tests', 'examples']),
python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*',
zip_safe = False,
classifiers = [
'Environment :: Console',
'Framework :: Ansible',
'Intended Audience :: System Administrators',
'License :: OSI Approved :: BSD License',
'Operating System :: MacOS :: MacOS X',
'Operating System :: POSIX',
'Programming Language :: Python',
'Programming Language :: Python :: 2.4',
'Programming Language :: Python :: 2.5',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: 3.11',
'Programming Language :: Python :: 3.12',
'Programming Language :: Python :: Implementation :: CPython',
'Topic :: System :: Distributed Computing',
'Topic :: System :: Systems Administration',
......
mitogen-0.3.7/tests/testlib.py 0 → 100644
View file @ 985463fd
import errno
import io
import logging
import os
import random
import re
import socket
import stat
import sys
import threading
import time
import traceback
import unittest
try:
import configparser
except ImportError:
import ConfigParser as configparser
import psutil
if sys.version_info < (3, 0):
import subprocess32 as subprocess
else:
import subprocess
import mitogen.core
import mitogen.fork
import mitogen.master
import mitogen.utils
try:
import faulthandler
except ImportError:
faulthandler = None
try:
import urlparse
except ImportError:
import urllib.parse as urlparse
try:
from cStringIO import StringIO
except ImportError:
from io import StringIO
try:
BaseException
except NameError:
BaseException = Exception
LOG = logging.getLogger(__name__)
TESTS_DIR = os.path.join(os.path.dirname(__file__))
ANSIBLE_LIB_DIR = os.path.join(TESTS_DIR, 'ansible', 'lib')
ANSIBLE_MODULE_UTILS_DIR = os.path.join(TESTS_DIR, 'ansible', 'lib', 'module_utils')
ANSIBLE_MODULES_DIR = os.path.join(TESTS_DIR, 'ansible', 'lib', 'modules')
DATA_DIR = os.path.join(TESTS_DIR, 'data')
MODS_DIR = os.path.join(TESTS_DIR, 'data', 'importer')
sys.path.append(DATA_DIR)
sys.path.append(MODS_DIR)
if mitogen.is_master:
mitogen.utils.log_to_file()
if faulthandler is not None:
faulthandler.enable()
#
# Temporary hack: Operon changed logging somewhat, and this broke LogCapturer /
# log_handler_test.
#
mitogen.core.LOG.propagate = True
def base_executable(executable=None):
'''Return the path of the Python executable used to create the virtualenv.
'''
# https://docs.python.org/3/library/venv.html
# https://github.com/pypa/virtualenv/blob/main/src/virtualenv/discovery/py_info.py
# https://virtualenv.pypa.io/en/16.7.9/reference.html#compatibility-with-the-stdlib-venv-module
if executable is None:
executable = sys.executable
if not executable:
raise ValueError
try:
base_executable = sys._base_executable
except AttributeError:
base_executable = None
if base_executable and base_executable != executable:
return base_executable
# Python 2.x only has sys.base_prefix if running outside a virtualenv.
try:
sys.base_prefix
except AttributeError:
# Python 2.x outside a virtualenv
return executable
# Python 3.3+ has sys.base_prefix. In a virtualenv it differs to sys.prefix.
if sys.base_prefix == sys.prefix:
return executable
while executable.startswith(sys.prefix) and stat.S_ISLNK(os.lstat(executable).st_mode):
dirname = os.path.dirname(executable)
target = os.path.join(dirname, os.readlink(executable))
executable = os.path.abspath(os.path.normpath(target))
print(executable)
if executable.startswith(sys.base_prefix):
return executable
# Virtualenvs record details in pyvenv.cfg
parser = configparser.RawConfigParser()
with io.open(os.path.join(sys.prefix, 'pyvenv.cfg'), encoding='utf-8') as f:
content = u'[virtualenv]\n' + f.read()
try:
parser.read_string(content)
except AttributeError:
parser.readfp(io.StringIO(content))
# virtualenv style pyvenv.cfg includes the base executable.
# venv style pyvenv.cfg doesn't.
try:
return parser.get(u'virtualenv', u'base-executable')
except configparser.NoOptionError:
pass
basename = os.path.basename(executable)
home = parser.get(u'virtualenv', u'home')
return os.path.join(home, basename)
def data_path(suffix):
path = os.path.join(DATA_DIR, suffix)
if path.endswith('.key'):
# SSH is funny about private key permissions.
os.chmod(path, int('0600', 8))
return path
def threading__thread_is_alive(thread):
"""Return whether the thread is alive (Python version compatibility shim).
On Python >= 3.8 thread.isAlive() is deprecated (removed in Python 3.9).
On Python <= 2.5 thread.is_alive() isn't present (added in Python 2.6).
"""
try:
return thread.is_alive()
except AttributeError:
return thread.isAlive()
def threading_thread_name(thread):
try:
return thread.name # Available in Python 2.6+
except AttributeError:
return thread.getName() # Deprecated in Python 3.10+
def wait_for_port(
host,
port,
pattern=None,
connect_timeout=0.5,
receive_timeout=0.5,
overall_timeout=5.0,
sleep=0.1,
):
"""Attempt to connect to host/port, for upto overall_timeout seconds.
If a regex pattern is supplied try to find it in the initial data.
Return None on success, or raise on error.
"""
start = mitogen.core.now()
end = start + overall_timeout
addr = (host, port)
while mitogen.core.now() < end:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(connect_timeout)
try:
sock.connect(addr)
except socket.error:
# Failed to connect. So wait then retry.
time.sleep(sleep)
continue
if not pattern:
# Success: We connected & there's no banner check to perform.
sock.shutdown(socket.SHUT_RDWR)
sock.close()
return
sock.settimeout(receive_timeout)
data = mitogen.core.b('')
found = False
while mitogen.core.now() < end:
try:
resp = sock.recv(1024)
except socket.timeout:
# Server stayed up, but had no data. Retry the recv().
continue
if not resp:
# Server went away. Wait then retry the connection.
time.sleep(sleep)
break
data += resp
if re.search(mitogen.core.b(pattern), data):
found = True
break
try:
sock.shutdown(socket.SHUT_RDWR)
except socket.error:
e = sys.exc_info()[1]
# On Mac OS X - a BSD variant - the above code only succeeds if the
# operating system thinks that the socket is still open when
# shutdown() is invoked. If Python is too slow and the FIN packet
# arrives before that statement can be reached, then OS X kills the
# sock.shutdown() statement with:
#
# socket.error: [Errno 57] Socket is not connected
#
# Protect shutdown() with a try...except that catches the
# socket.error, test to make sure Errno is right, and ignore it if
# Errno matches.
if e.errno == 57:
pass
else:
raise
sock.close()
if found:
# Success: We received the banner & found the desired pattern
return
else:
# Failure: The overall timeout expired
if pattern:
raise socket.timeout('Timed out while searching for %r from %s:%s'
% (pattern, host, port))
else:
raise socket.timeout('Timed out while connecting to %s:%s'
% (host, port))
def sync_with_broker(broker, timeout=10.0):
"""
Insert a synchronization barrier between the calling thread and the Broker
thread, ensuring it has completed at least one full IO loop before
returning.
Used to block while asynchronous stuff (like defer()) happens on the
broker.
"""
sem = mitogen.core.Latch()
broker.defer(sem.put, None)
sem.get(timeout=timeout)
def log_fd_calls():
mypid = os.getpid()
l = threading.Lock()
real_pipe = os.pipe
def pipe():
l.acquire()
try:
rv = real_pipe()
if mypid == os.getpid():
sys.stdout.write('\n%s\n' % (rv,))
traceback.print_stack(limit=3)
sys.stdout.write('\n')
return rv
finally:
l.release()
os.pipe = pipe
real_socketpair = socket.socketpair
def socketpair(*args):
l.acquire()
try:
rv = real_socketpair(*args)
if mypid == os.getpid():
sys.stdout.write('\n%s -> %s\n' % (args, rv))
traceback.print_stack(limit=3)
sys.stdout.write('\n')
return rv
finally:
l.release()
socket.socketpair = socketpair
real_dup2 = os.dup2
def dup2(*args):
l.acquire()
try:
real_dup2(*args)
if mypid == os.getpid():
sys.stdout.write('\n%s\n' % (args,))
traceback.print_stack(limit=3)
sys.stdout.write('\n')
finally:
l.release()
os.dup2 = dup2
real_dup = os.dup
def dup(*args):
l.acquire()
try:
rv = real_dup(*args)
if mypid == os.getpid():
sys.stdout.write('\n%s -> %s\n' % (args, rv))
traceback.print_stack(limit=3)
sys.stdout.write('\n')
return rv
finally:
l.release()
os.dup = dup
class CaptureStreamHandler(logging.StreamHandler):
def __init__(self, *args, **kwargs):
logging.StreamHandler.__init__(self, *args, **kwargs)
self.msgs = []
def emit(self, msg):
self.msgs.append(msg)
logging.StreamHandler.emit(self, msg)
class LogCapturer(object):
def __init__(self, name=None):
self.sio = StringIO()
self.logger = logging.getLogger(name)
self.handler = CaptureStreamHandler(self.sio)
self.old_propagate = self.logger.propagate
self.old_handlers = self.logger.handlers
self.old_level = self.logger.level
def start(self):
self.logger.handlers = [self.handler]
self.logger.propagate = False
self.logger.level = logging.DEBUG
def raw(self):
s = self.sio.getvalue()
# Python 2.x logging package hard-wires UTF-8 output.
if isinstance(s, mitogen.core.BytesType):
s = s.decode('utf-8')
return s
def msgs(self):
return self.handler.msgs
def __enter__(self):
self.start()
return self
def __exit__(self, _1, _2, _3):
self.stop()
def stop(self):
self.logger.level = self.old_level
self.logger.handlers = self.old_handlers
self.logger.propagate = self.old_propagate
return self.raw()
class TestCase(unittest.TestCase):
@classmethod
def setUpClass(cls):
# This is done in setUpClass() so we have a chance to run before any
# Broker() instantiations in setUp() etc.
mitogen.fork.on_fork()
cls._fds_before = psutil.Process().open_files()
# Ignore children started by external packages - in particular
# multiprocessing.resource_tracker.main()`, started when some Ansible
# versions instantiate a `multithreading.Lock()`.
cls._children_before = frozenset(psutil.Process().children())
super(TestCase, cls).setUpClass()
ALLOWED_THREADS = set([
'MainThread',
'mitogen.master.join_thread_async'
])
def _teardown_check_threads(self):
counts = {}
for thread in threading.enumerate():
name = threading_thread_name(thread)
# Python 2.4: enumerate() may return stopped threads.
assert \
not threading__thread_is_alive(thread) \
or name in self.ALLOWED_THREADS, \
'Found thread %r still running after tests.' % (name,)
counts[name] = counts.get(name, 0) + 1
for name in counts:
assert counts[name] == 1, \
'Found %d copies of thread %r running after tests.' % (
counts[name], name
)
def _teardown_check_fds(self):
mitogen.core.Latch._on_fork()
fds_after = psutil.Process().open_files()
fds_leaked = len(self._fds_before) != len(fds_after)
if not fds_leaked:
return
else:
if sys.platform == 'linux':
subprocess.check_call(
'lsof +E -w -p %i | grep -vw mem' % (os.getpid(),),
shell=True,
)
else:
subprocess.check_call(
'lsof -w -p %i | grep -vw mem' % (os.getpid(),),
shell=True,
)
assert 0, "%s leaked FDs: %s\nBefore:\t%s\nAfter:\t%s" % (
self, fds_leaked, self._fds_before, fds_after,
)
# Some class fixtures (like Ansible MuxProcess) start persistent children
# for the duration of the class.
no_zombie_check = False
def _teardown_check_zombies(self):
if self.no_zombie_check:
return
# pid=0: Wait for any child process in the same process group as us.
# WNOHANG: Don't block if no processes ready to report status.
try:
pid, status = os.waitpid(0, os.WNOHANG)
except OSError as e:
# ECHILD: there are no child processes in our group.
if e.errno == errno.ECHILD:
return
raise
if pid:
assert 0, "%s failed to reap subprocess %d (status %d)." % (
self, pid, status
)
children_after = frozenset(psutil.Process().children())
children_leaked = children_after.difference(self._children_before)
if not children_leaked:
return
print('Leaked children of unit test process:')
subprocess.check_call(
['ps', '-o', 'user,pid,%cpu,%mem,vsz,rss,tty,stat,start,time,command', '-ww', '-p',
','.join(str(p.pid) for p in children_leaked),
],
)
if self._children_before:
print('Pre-existing children of unit test process:')
subprocess.check_call(
['ps', '-o', 'user,pid,%cpu,%mem,vsz,rss,tty,stat,start,time,command', '-ww', '-p',
','.join(str(p.pid) for p in self._children_before),
],
)
assert 0, "%s leaked still-running subprocesses." % (self,)
def tearDown(self):
self._teardown_check_zombies()
self._teardown_check_threads()
self._teardown_check_fds()
super(TestCase, self).tearDown()
def assertRaises(self, exc, func, *args, **kwargs):
"""Like regular assertRaises, except return the exception that was
raised. Can't use context manager because tests must run on Python2.4"""
try:
func(*args, **kwargs)
except exc:
e = sys.exc_info()[1]
return e
except BaseException:
LOG.exception('Original exception')
e = sys.exc_info()[1]
assert 0, '%r raised %r, not %r' % (func, e, exc)
assert 0, '%r did not raise %r' % (func, exc)
def get_docker_host():
url = os.environ.get('DOCKER_HOST')
if url in (None, 'http+docker://localunixsocket'):
return 'localhost'
parsed = urlparse.urlparse(url)
return parsed.netloc.partition(':')[0]
class DockerizedSshDaemon(object):
PORT_RE = re.compile(
# e.g. 0.0.0.0:32771, :::32771, [::]:32771'
r'(?P<addr>[0-9.]+|::|\[[a-f0-9:.]+\]):(?P<port>[0-9]+)',
)
@classmethod
def get_port(cls, container):
s = subprocess.check_output(['docker', 'port', container, '22/tcp'])
m = cls.PORT_RE.search(s.decode())
if not m:
raise ValueError('could not find SSH port in: %r' % (s,))
return int(m.group('port'))
def start_container(self):
try:
subprocess.check_output(['docker', '--version'])
except Exception:
raise unittest.SkipTest('Docker binary is unavailable')
self.container_name = 'mitogen-test-%08x' % (random.getrandbits(64),)
args = [
'docker',
'run',
'--detach',
'--privileged',
'--publish-all',
'--name', self.container_name,
self.image,
]
subprocess.check_output(args)
def __init__(self, mitogen_test_distro=os.environ.get('MITOGEN_TEST_DISTRO', 'debian9')):
if '-' in mitogen_test_distro:
distro, _py3 = mitogen_test_distro.split('-')
else:
distro = mitogen_test_distro
_py3 = None
if _py3 == 'py3':
self.python_path = '/usr/bin/python3'
else:
self.python_path = '/usr/bin/python'
self.image = 'public.ecr.aws/n5z0e8q9/%s-test' % (distro,)
self.start_container()
self.host = self.get_host()
self.port = self.get_port(self.container_name)
def get_host(self):
return get_docker_host()
def wait_for_sshd(self):
wait_for_port(self.get_host(), self.port, pattern='OpenSSH')
def check_processes(self):
args = ['docker', 'exec', self.container_name, 'ps', '-o', 'comm=']
counts = {}
for comm in subprocess.check_output(args).decode().splitlines():
comm = comm.strip()
counts[comm] = counts.get(comm, 0) + 1
if counts != {'ps': 1, 'sshd': 1}:
assert 0, (
'Docker container %r contained extra running processes '
'after test completed: %r' % (
self.container_name,
counts
)
)
def close(self):
args = ['docker', 'rm', '-f', self.container_name]
subprocess.check_output(args)
class BrokerMixin(object):
broker_class = mitogen.master.Broker
broker_shutdown = False
def setUp(self):
super(BrokerMixin, self).setUp()
self.broker = self.broker_class()
def tearDown(self):
if not self.broker_shutdown:
self.broker.shutdown()
self.broker.join()
del self.broker
super(BrokerMixin, self).tearDown()
def sync_with_broker(self):
sync_with_broker(self.broker)
class RouterMixin(BrokerMixin):
router_class = mitogen.master.Router
def setUp(self):
super(RouterMixin, self).setUp()
self.router = self.router_class(self.broker)
def tearDown(self):
del self.router
super(RouterMixin, self).tearDown()
class DockerMixin(RouterMixin):
@classmethod
def setUpClass(cls):
super(DockerMixin, cls).setUpClass()
if os.environ.get('SKIP_DOCKER_TESTS'):
raise unittest.SkipTest('SKIP_DOCKER_TESTS is set')
# we want to be able to override test distro for some tests that need a different container spun up
daemon_args = {}
if hasattr(cls, 'mitogen_test_distro'):
daemon_args['mitogen_test_distro'] = cls.mitogen_test_distro
cls.dockerized_ssh = DockerizedSshDaemon(**daemon_args)
cls.dockerized_ssh.wait_for_sshd()
@classmethod
def tearDownClass(cls):
cls.dockerized_ssh.check_processes()
cls.dockerized_ssh.close()
super(DockerMixin, cls).tearDownClass()
@property
def docker_ssh_default_kwargs(self):
return {
'hostname': self.dockerized_ssh.host,
'port': self.dockerized_ssh.port,
'check_host_keys': 'ignore',
'ssh_debug_level': 3,
# https://www.openssh.com/legacy.html
# ssh-rsa uses SHA1. Least worst available with CentOS 7 sshd.
# Rejected by default in newer ssh clients (e.g. Ubuntu 22.04).
# Duplicated cases in
# - tests/ansible/ansible.cfg
# - tests/ansible/integration/connection_delegation/delegate_to_template.yml
# - tests/ansible/integration/connection_delegation/stack_construction.yml
# - tests/ansible/integration/process/unix_socket_cleanup.yml
# - tests/ansible/integration/ssh/variables.yml
# - tests/testlib.py
'ssh_args': [
'-o', 'HostKeyAlgorithms +ssh-rsa',
'-o', 'PubkeyAcceptedKeyTypes +ssh-rsa',
],
'python_path': self.dockerized_ssh.python_path,
}
def docker_ssh(self, **kwargs):
for k, v in self.docker_ssh_default_kwargs.items():
kwargs.setdefault(k, v)
return self.router.ssh(**kwargs)
def docker_ssh_any(self, **kwargs):
return self.docker_ssh(
username='mitogen__has_sudo_nopw',
password='has_sudo_nopw_password',
)
nextgen-router @ b1b259b4
Subproject commit f3c29e8153470a8e5d903e679cc82d9a4dd26ae5
Subproject commit b1b259b455f2c41892ab8b5b996116f71cb0fd27
init @ 734d8607
Subproject commit eaa2b3a15f318a9ad3d68845d0c630fd4b6bf5ff
Subproject commit 734d86077fa83b1d78a695327760347d3649daa3
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment